Mailman 3 May 2013 - Python-Dev

PEP 409 and the stdlib
by Ethan Furman 21 Jun '13

21 Jun '13

As a quick reminder, PEP 409 allows this: try: ... except AnError: raise SomeOtherError from None so that if the exception is not caught, we get the traditional single exception traceback, instead of the new: During handling of the above exception, another exception occurred My question: How do we go about putting this in the stdlib? Is this one of the occasions where we don't do it unless we're modifying a module already for some other reason? For that matter, should we? Pros: Makes tracebacks much less confusing, especially coming from a library Cons: Could hide bugs unrelated to what is being caught and transformed -- ~Ethan~

12 32

Ordering keyword dicts
by Guido van Rossum 10 Jun '13

10 Jun '13

On Sat, May 18, 2013 at 10:27 PM, Raymond Hettinger <raymond.hettinger(a)gmail.com> wrote: > BTW, I'm +1 on the idea for ordering keyword-args. It makes > it easier to debug if the arguments show-up in the order they > were created. AFAICT, no purpose is served by scrambling them > (which is exacerbated by the new randomized hashing security feature). I'm slow at warming up to the idea. My main concern is speed -- since most code doesn't need it and function calls are already slow (and obviously very common :-) it would be a shame if this slowed down function calls that don't need it noticeably. An observation is that it's only necessary to preserve order if the function definition uses **kwds. AFAIK we currently don't know if this is the case when the call is made though, but perhaps the information could be made available to the call site somehow. There are also many special cases to consider; e.g. using **kwds in the call where kwds is an unordered dict, or calls from C, or calls to C. But maybe someone considers this a challenge and comes up with a patch? The benefits to *some* use cases would be obvious. -- --Guido van Rossum (python.org/~guido)

14 15

PEP 442: Safe object finalization
by Antoine Pitrou 03 Jun '13

03 Jun '13

Hello, I would like to submit the following PEP for discussion and evaluation. Regards Antoine. PEP: 442 Title: Safe object finalization Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou <solipsis(a)pitrou.net> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2013-05-18 Python-Version: 3.4 Post-History: Resolution: TBD Abstract ======== This PEP proposes to deal with the current limitations of object finalization. The goal is to be able to define and run finalizers for any object, regardless of their position in the object graph. This PEP doesn't call for any change in Python code. Objects with existing finalizers will benefit automatically. Definitions =========== Reference A directional link from an object to another. The target of the reference is kept alive by the reference, as long as the source is itself alive and the reference isn't cleared. Weak reference A directional link from an object to another, which doesn't keep alive its target. This PEP focusses on non-weak references. Reference cycle A cyclic subgraph of directional links between objects, which keeps those objects from being collected in a pure reference-counting scheme. Cyclic isolate (CI) A reference cycle in which no object is referenced from outside the cycle *and* whose objects are still in a usable, non-broken state: they can access each other from their respective finalizers. Cyclic garbage collector (GC) A device able to detect cyclic isolates and turn them into cyclic trash. Objects in cyclic trash are eventually disposed of by the natural effect of the references being cleared and their reference counts dropping to zero. Cyclic trash (CT) A reference cycle, or former reference cycle, in which no object is referenced from outside the cycle *and* whose objects have started being cleared by the GC. Objects in cyclic trash are potential zombies; if they are accessed by Python code, the symptoms can vary from weird AttributeErrors to crashes. Zombie / broken object An object part of cyclic trash. The term stresses that the object is not safe: its outgoing references may have been cleared, or one of the objects it references may be zombie. Therefore, it should not be accessed by arbitrary code (such as finalizers). Finalizer A function or method called when an object is intended to be disposed of. The finalizer can access the object and release any resource held by the object (for example mutexes or file descriptors). An example is a ``__del__`` method. Resurrection The process by which a finalizer creates a new reference to an object in a CI. This can happen as a quirky but supported side-effect of ``__del__`` methods. Impact ====== While this PEP discusses CPython-specific implementation details, the change in finalization semantics is expected to affect the Python ecosystem as a whole. In particular, this PEP obsoletes the current guideline that "objects with a ``__del__`` method should not be part of a reference cycle". Benefits ======== The primary benefits of this PEP regard objects with finalizers, such as objects with a ``__del__`` method and generators with a ``finally`` block. Those objects can now be reclaimed when they are part of a reference cycle. The PEP also paves the way for further benefits: * The module shutdown procedure may not need to set global variables to None anymore. This could solve a well-known class of irritating issues. The PEP doesn't change the semantics of: * Weak references caught in reference cycles. * C extension types with a custom ``tp_dealloc`` function. Description =========== Reference-counted disposal -------------------------- In normal reference-counted disposal, an object's finalizer is called just before the object is deallocated. If the finalizer resurrects the object, deallocation is aborted. *However*, if the object was already finalized, then the finalizer isn't called. This prevents us from finalizing zombies (see below). Disposal of cyclic isolates --------------------------- Cyclic isolates are first detected by the garbage collector, and then disposed of. The detection phase doesn't change and won't be described here. Disposal of a CI traditionally works in the following order: 1. Weakrefs to CI objects are cleared, and their callbacks called. At this point, the objects are still safe to use. 2. The CI becomes a CT as the GC systematically breaks all known references inside it (using the ``tp_clear`` function). 3. Nothing. All CT objects should have been disposed of in step 2 (as a side-effect of clearing references); this collection is finished. This PEP proposes to turn CI disposal into the following sequence (new steps are in bold): 1. Weakrefs to CI objects are cleared, and their callbacks called. At this point, the objects are still safe to use. 2. **The finalizers of all CI objects are called.** 3. **The CI is traversed again to determine if it is still isolated. If it is determined that at least one object in CI is now reachable from outside the CI, this collection is aborted and the whole CI is resurrected. Otherwise, proceed.** 4. The CI becomes a CT as the GC systematically breaks all known references inside it (using the ``tp_clear`` function). 5. Nothing. All CT objects should have been disposed of in step 4 (as a side-effect of clearing references); this collection is finished. C-level changes =============== Type objects get a new ``tp_finalize`` slot to which ``__del__`` methods are bound. Generators are also modified to use this slot, rather than ``tp_del``. At the C level, a ``tp_finalize`` function is a normal function which will be called with a regular, alive object as its only argument. It should not attempt to revive or collect the object. For compatibility, ``tp_del`` is kept in the type structure. Handling of objects with a non-NULL ``tp_del`` is unchanged: when part of a CI, they are not finalized and end up in ``gc.garbage``. However, a non-NULL ``tp_del`` is not encountered anymore in the CPython source tree (except for testing purposes). On the internal side, a bit is reserved in the GC header for GC-managed objects to signal that they were finalized. This helps avoid finalizing an object twice (and, especially, finalizing a CT object after it was broken by the GC). Discussion ========== Predictability -------------- Following this scheme, an object's finalizer is always called exactly once. The only exception is if an object is resurrected: the finalizer will be called again later. For CI objects, the order in which finalizers are called (step 2 above) is undefined. Safety ------ It is important to explain why the proposed change is safe. There are two aspects to be discussed: * Can a finalizer access zombie objects (including the object being finalized)? * What happens if a finalizer mutates the object graph so as to impact the CI? Let's discuss the first issue. We will divide possible cases in two categories: * If the object being finalized is part of the CI: by construction, no objects in CI are zombies yet, since CI finalizers are called before any reference breaking is done. Therefore, the finalizer cannot access zombie objects, which don't exist. * If the object being finalized is not part of the CI/CT: by definition, objects in the CI/CT don't have any references pointing to them from outside the CI/CT. Therefore, the finalizer cannot reach any zombie object (that is, even if the object being finalized was itself referenced from a zombie object). Now for the second issue. There are three potential cases: * The finalizer clears an existing reference to a CI object. The CI object may be disposed of before the GC tries to break it, which is fine (the GC simply has to be aware of this possibility). * The finalizer creates a new reference to a CI object. This can only happen from a CI object's finalizer (see above why). Therefore, the new reference will be detected by the GC after all CI finalizers are called (step 3 above), and collection will be aborted without any objects being broken. * The finalizer clears or creates a reference to a non-CI object. By construction, this is not a problem. Implementation ============== An implementation is available in branch ``finalize`` of the repository at http://hg.python.org/features/finalize/. Validation ========== Besides running the normal Python test suite, the implementation adds test cases for various finalization possibilities including reference cycles, object resurrection and legacy ``tp_del`` slots. The implementation has also been checked to not produce any regressions on the following test suites: * `Tulip <http://code.google.com/p/tulip/>`_, which makes an extensive use of generators * `Tornado <http://www.tornadoweb.org>`_ * `SQLAlchemy <http://www.sqlalchemy.org/>`_ * `Django <https://www.djangoproject.com/>`_ * `zope.interface <http://pypi.python.org/pypi/zope.interface>`_ References ========== Notes about reference cycle collection and weak reference callbacks: http://hg.python.org/cpython/file/4e687d53b645/Modules/gc_weakref.txt Generator memory leak: http://bugs.python.org/issue17468 Allow objects to decide if they can be collected by GC: http://bugs.python.org/issue9141 Module shutdown procedure based on GC http://bugs.python.org/issue812369 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

8 21

Problem building Python 2.7.5 with separate sysroot
by Paul Smith 02 Jun '13

02 Jun '13

Hi all. I'm trying to build Python 2.7.5 on a GNU/Linux (Linux Mint 14) system, but using a different sysroot (that is, a separate <d>/usr/include, <d>/usr/lib, etc., not the real one for my system). I have shell script wrappers around GCC and its various tools that invoke it with the right paths to force this to happen, and when I call Python's configure I send along "CC=sysroot-gcc", etc. for all the various tools. Note that it's not really a cross-compilation because the target is also a GNU/Linux system on the same hardware architecture. The majority of Python builds just fine like this. However, I'm having serious problems building modules such as fcntl, etc. Looking at the output from the makefile, I can see that somehow, someone is forcibly adding "-I/usr/include/x86_64-linux-gnu" to the link line: building 'termios' extension sysroot-gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I/usr/include/x86_64-linux-gnu -I/common/sysroot/tools/usr/include -I/common/sysroot/tools/usr/include-fixed -I/usr/local/include -I/home/workspaces/psmith/python/obj/src/Python-2.7.5/Include -I/home/workspaces/psmith/python/obj/bld/python -c /home/workspaces/psmith/python/obj/src/Python-2.7.5/Modules/termios.c -o build/temp.linux-x86_64-2.7/home/workspaces/psmith/python/obj/src/Python-2.7.5/Modules/termios.o This fails miserably because the headers in /usr/include/x86_64-linux-gnu do not play at all nicely with my other sysroot headers. Ditto for other extensions like fcntl, etc. I've searched high and low in the Python source, generated makefiles, config.log, etc. and I cannot find where this -I flag is coming from anywhere. I found the --oldincludedir flag to configure and set it to point into my sysroot as well, but that didn't help: the /usr/include still appears when building these extensions. Can anyone tell me where Python is getting these -I flags and what I need to do to tell it to NOT use those flags when building extensions? I'd also like to remove the -I/usr/local/include, although this is not actually causing me problems right now.

3 4

Re: [Python-Dev] [Python-checkins] cpython: Add a reset_name argument to importlib.util.module_to_load in order to
by Brett Cannon 31 May '13

31 May '13

I realize this broke the buildbots. Missed part of a diff in the commit. I'm trying to split a massive CL into reasonable commit sizes, so please be patient. On Fri, May 31, 2013 at 6:11 PM, brett.cannon <python-checkins(a)python.org>wrote: > http://hg.python.org/cpython/rev/39cc1b04713e > changeset: 83998:39cc1b04713e > user: Brett Cannon <brett(a)python.org> > date: Fri May 31 18:11:17 2013 -0400 > summary: > Add a reset_name argument to importlib.util.module_to_load in order to > control whether to reset the module's __name__ attribute in case a > reload is being done. > > files: > Doc/library/importlib.rst | 6 +++++- > Lib/importlib/_bootstrap.py | 14 +++++++++++++- > Lib/test/test_importlib/test_util.py | 12 ++++++++++++ > 3 files changed, 30 insertions(+), 2 deletions(-) > > > diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst > --- a/Doc/library/importlib.rst > +++ b/Doc/library/importlib.rst > @@ -788,7 +788,7 @@ > > .. versionadded:: 3.3 > > -.. function:: module_to_load(name) > +.. function:: module_to_load(name, *, reset_name=True) > > Returns a :term:`context manager` which provides the module to load. > The > module will either come from :attr:`sys.modules` in the case of > reloading or > @@ -796,6 +796,10 @@ > :attr:`sys.modules` occurs if the module was new and an exception was > raised. > > + If **reset_name** is true and the module requested is being reloaded > then > + the module's :attr:`__name__` attribute will > + be reset to **name**, else it will be left untouched. > + > .. versionadded:: 3.4 > > .. decorator:: module_for_loader > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py > +++ b/Lib/importlib/_bootstrap.py > @@ -493,8 +493,14 @@ > > """ > > - def __init__(self, name): > + def __init__(self, name, *, reset_name=True): > + """Prepare the context manager. > + > + The reset_name argument specifies whether to unconditionally reset > + the __name__ attribute if the module is found to be a reload. > + """ > self._name = name > + self._reset_name = reset_name > > def __enter__(self): > self._module = sys.modules.get(self._name) > @@ -508,6 +514,12 @@ > # (otherwise an optimization shortcut in import.c becomes > wrong) > self._module.__initializing__ = True > sys.modules[self._name] = self._module > + elif self._reset_name: > + try: > + self._module.__name__ = self._name > + except AttributeError: > + pass > + > return self._module > > def __exit__(self, *args): > diff --git a/Lib/test/test_importlib/test_util.py > b/Lib/test/test_importlib/test_util.py > --- a/Lib/test/test_importlib/test_util.py > +++ b/Lib/test/test_importlib/test_util.py > @@ -55,6 +55,18 @@ > else: > self.fail('importlib.util.module_to_load swallowed an > exception') > > + def test_reset_name(self): > + # If reset_name is true then module.__name__ = name, else leave > it be. > + odd_name = 'not your typical name' > + created_module = imp.new_module(self.module_name) > + created_module.__name__ = odd_name > + sys.modules[self.module_name] = created_module > + with util.module_to_load(self.module_name) as module: > + self.assertEqual(module.__name__, self.module_name) > + created_module.__name__ = odd_name > + with util.module_to_load(self.module_name, reset_name=False) as > module: > + self.assertEqual(module.__name__, odd_name) > + > > class ModuleForLoaderTests(unittest.TestCase): > > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > >

1 0

Summary of Python tracker Issues
by Python tracker 31 May '13

31 May '13

ACTIVITY SUMMARY (2013-05-24 - 2013-05-31) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3997 (+25) closed 25884 (+34) total 29881 (+59) Open issues with patches: 1780 Issues opened (45) ================== #6386: importing yields unexpected results when initial script is a s http://bugs.python.org/issue6386 reopened by ncoghlan #17269: getaddrinfo segfaults on OS X when provided with invalid argum http://bugs.python.org/issue17269 reopened by ned.deily #18050: embedded interpreter or virtualenv fails with "ImportError: ca http://bugs.python.org/issue18050 opened by samueljohn #18052: IDLE 3.3.2 Windows taskbar icon regression http://bugs.python.org/issue18052 opened by terry.reedy #18053: Add checks for Misc/NEWS in make patchcheck http://bugs.python.org/issue18053 opened by Yogesh.Chaudhari #18054: Add more exception related assertions to unittest http://bugs.python.org/issue18054 opened by ncoghlan #18055: Stop using imp in IDLE http://bugs.python.org/issue18055 opened by brett.cannon #18056: Document importlib._bootstrap.NamespaceLoader http://bugs.python.org/issue18056 opened by brett.cannon #18057: Register NamespaceLoader with importlib.abc.Loader http://bugs.python.org/issue18057 opened by brett.cannon #18058: Define is_package for NamespaceLoader http://bugs.python.org/issue18058 opened by brett.cannon #18059: Add multibyte encoding support to pyexpat http://bugs.python.org/issue18059 opened by serhiy.storchaka #18060: Updating _fields_ of a derived struct type yields a bad cif http://bugs.python.org/issue18060 opened by lauri.alanko #18061: m68k Python 3.3 test results http://bugs.python.org/issue18061 opened by mirabilos #18062: m68k FPU precision issue http://bugs.python.org/issue18062 opened by mirabilos #18064: IDLE: add current directory to open_module http://bugs.python.org/issue18064 opened by terry.reedy #18065: set __path__ = [] for frozen packages http://bugs.python.org/issue18065 opened by brett.cannon #18066: Remove SGI-specific code from pty.py http://bugs.python.org/issue18066 opened by akuchling #18068: pickle + weakref.proxy(self) http://bugs.python.org/issue18068 opened by phd #18069: Subprocess picks the wrong executable on Windows http://bugs.python.org/issue18069 opened by berdario #18071: _osx_support compiler_fixup http://bugs.python.org/issue18071 opened by samueljohn #18073: pickle.Unpickler may read too many bytes, causing hangs with b http://bugs.python.org/issue18073 opened by jm #18076: Implement importlib.util.decode_source_bytes() http://bugs.python.org/issue18076 opened by brett.cannon #18078: threading.Condition to allow notify on a specific waiter http://bugs.python.org/issue18078 opened by JBernardo #18081: test_logging failure in WarningsTest on buildbots http://bugs.python.org/issue18081 opened by r.david.murray #18082: Inconsistent behavior of IOBase methods on closed files http://bugs.python.org/issue18082 opened by dwight.guth #18083: _sysconfigdata.py is installed in an arch-independent director http://bugs.python.org/issue18083 opened by automatthias #18085: Verifying refcounts.dat http://bugs.python.org/issue18085 opened by serhiy.storchaka #18088: Create importlib.abc.Loader.init_module_attrs() http://bugs.python.org/issue18088 opened by brett.cannon #18089: Create importlib.abc.InspectLoader.load_module() http://bugs.python.org/issue18089 opened by brett.cannon #18090: dict_contains first argument declared register, and shouldn't http://bugs.python.org/issue18090 opened by larry #18091: Remove PyNoArgsFunction http://bugs.python.org/issue18091 opened by larry #18092: Python 2.7.5 installation broken on OpenSuse 12.2 http://bugs.python.org/issue18092 opened by Andreas.Jung #18093: Move main functions to a separate Programs directory http://bugs.python.org/issue18093 opened by ncoghlan #18094: Skip tests in test_uuid not silently http://bugs.python.org/issue18094 opened by serhiy.storchaka #18095: unable to invoke socket.connect with AF_UNSPEC http://bugs.python.org/issue18095 opened by Roman.Valov #18096: bad library order returned by python-config.in http://bugs.python.org/issue18096 opened by taylor #18099: wsgiref sets Content-Length: 0 on 304 Not Modified http://bugs.python.org/issue18099 opened by flox #18100: socket.sendall() cannot send buffers of 2GB or more http://bugs.python.org/issue18100 opened by Tom.van.Leeuwen #18101: Tk.split() doesn't work with nested Unicode strings http://bugs.python.org/issue18101 opened by serhiy.storchaka #18102: except-clause with own exception class inside generator can le http://bugs.python.org/issue18102 opened by hagen #18103: Create a GUI test framework for Idle http://bugs.python.org/issue18103 opened by terry.reedy #18104: Idle: make human-mediated GUI tests usable http://bugs.python.org/issue18104 opened by terry.reedy #18105: ElementTree writes invalid files when UTF-16 encoding is speci http://bugs.python.org/issue18105 opened by Adam.Urban #18106: There are unused variables in Lib/test/test_collections.py http://bugs.python.org/issue18106 opened by vajrasky #18108: shutil.chown should support dir_fd and follow_symlinks keyword http://bugs.python.org/issue18108 opened by cjwatson Most recent 15 issues with no replies (15) ========================================== #18108: shutil.chown should support dir_fd and follow_symlinks keyword http://bugs.python.org/issue18108 #18101: Tk.split() doesn't work with nested Unicode strings http://bugs.python.org/issue18101 #18099: wsgiref sets Content-Length: 0 on 304 Not Modified http://bugs.python.org/issue18099 #18096: bad library order returned by python-config.in http://bugs.python.org/issue18096 #18095: unable to invoke socket.connect with AF_UNSPEC http://bugs.python.org/issue18095 #18094: Skip tests in test_uuid not silently http://bugs.python.org/issue18094 #18089: Create importlib.abc.InspectLoader.load_module() http://bugs.python.org/issue18089 #18088: Create importlib.abc.Loader.init_module_attrs() http://bugs.python.org/issue18088 #18082: Inconsistent behavior of IOBase methods on closed files http://bugs.python.org/issue18082 #18081: test_logging failure in WarningsTest on buildbots http://bugs.python.org/issue18081 #18076: Implement importlib.util.decode_source_bytes() http://bugs.python.org/issue18076 #18073: pickle.Unpickler may read too many bytes, causing hangs with b http://bugs.python.org/issue18073 #18068: pickle + weakref.proxy(self) http://bugs.python.org/issue18068 #18066: Remove SGI-specific code from pty.py http://bugs.python.org/issue18066 #18065: set __path__ = [] for frozen packages http://bugs.python.org/issue18065 Most recent 15 issues waiting for review (15) ============================================= #18106: There are unused variables in Lib/test/test_collections.py http://bugs.python.org/issue18106 #18101: Tk.split() doesn't work with nested Unicode strings http://bugs.python.org/issue18101 #18094: Skip tests in test_uuid not silently http://bugs.python.org/issue18094 #18093: Move main functions to a separate Programs directory http://bugs.python.org/issue18093 #18078: threading.Condition to allow notify on a specific waiter http://bugs.python.org/issue18078 #18066: Remove SGI-specific code from pty.py http://bugs.python.org/issue18066 #18059: Add multibyte encoding support to pyexpat http://bugs.python.org/issue18059 #18055: Stop using imp in IDLE http://bugs.python.org/issue18055 #18049: Re-enable threading test on OSX http://bugs.python.org/issue18049 #18045: get_python_version is not import in bdist_rpm.py http://bugs.python.org/issue18045 #18038: Unhelpful error message on invalid encoding specification http://bugs.python.org/issue18038 #18033: Example for Profile Module shows incorrect method http://bugs.python.org/issue18033 #18020: html.escape 10x slower than cgi.escape http://bugs.python.org/issue18020 #18015: python 2.7.5 fails to unpickle namedtuple pickled by 2.7.3 or http://bugs.python.org/issue18015 #18013: cgi.FieldStorage does not parse W3C sample http://bugs.python.org/issue18013 Top 10 most discussed issues (10) ================================= #16832: Expose cache validity checking support in ABCMeta http://bugs.python.org/issue16832 19 msgs #17987: test.support.captured_stderr, captured_stdin not documented http://bugs.python.org/issue17987 16 msgs #17947: Code, test, and doc review for PEP-0435 Enum http://bugs.python.org/issue17947 12 msgs #18062: m68k FPU precision issue http://bugs.python.org/issue18062 12 msgs #18078: threading.Condition to allow notify on a specific waiter http://bugs.python.org/issue18078 9 msgs #1693050: \w not helpful for non-Roman scripts http://bugs.python.org/issue1693050 9 msgs #5124: IDLE - pasting text doesn't delete selection http://bugs.python.org/issue5124 8 msgs #12641: Remove -mno-cygwin from distutils http://bugs.python.org/issue12641 8 msgs #18085: Verifying refcounts.dat http://bugs.python.org/issue18085 8 msgs #18090: dict_contains first argument declared register, and shouldn't http://bugs.python.org/issue18090 8 msgs Issues closed (34) ================== #8240: ssl.SSLSocket.write may fail on non-blocking sockets http://bugs.python.org/issue8240 closed by pitrou #15392: Create a unittest framework for IDLE http://bugs.python.org/issue15392 closed by terry.reedy #17206: Py_XDECREF() expands its argument multiple times http://bugs.python.org/issue17206 closed by python-dev #17272: request.full_url: unexpected results on assignment http://bugs.python.org/issue17272 closed by orsenthil #17700: Update Curses HOWTO for 3.4 http://bugs.python.org/issue17700 closed by akuchling #17746: test_shutil.TestWhich.test_non_matching_mode fails when runnin http://bugs.python.org/issue17746 closed by serhiy.storchaka #17768: _decimal: allow NUL fill character http://bugs.python.org/issue17768 closed by skrah #17953: sys.modules cannot be reassigned http://bugs.python.org/issue17953 closed by brett.cannon #18009: os.write.__doc__ is misleading http://bugs.python.org/issue18009 closed by python-dev #18011: Inconsistency between b32decode() documentation, docstring and http://bugs.python.org/issue18011 closed by serhiy.storchaka #18025: Buffer overflow in BufferedIOBase.readinto() http://bugs.python.org/issue18025 closed by serhiy.storchaka #18040: SIGINT catching regression on windows in 2.7 http://bugs.python.org/issue18040 closed by tim.golden #18046: Simplify and clarify logging internals http://bugs.python.org/issue18046 closed by python-dev #18047: Descriptors get invoked in old-style objects and classes http://bugs.python.org/issue18047 closed by rhettinger #18051: get_python_version undefined in bdist_rpm.py http://bugs.python.org/issue18051 closed by Craig.McDaniel #18063: m68k struct alignment issue vs. PyException_HEAD http://bugs.python.org/issue18063 closed by pitrou #18067: In considering the current directory for importing modules, Py http://bugs.python.org/issue18067 closed by r.david.murray #18070: change importlib.util.module_for_loader to unconditionally set http://bugs.python.org/issue18070 closed by brett.cannon #18072: Add an implementation of importlib.abc.InspectLoader.get_code http://bugs.python.org/issue18072 closed by brett.cannon #18074: argparse: Namespace can contain critical characters http://bugs.python.org/issue18074 closed by Sworddragon #18075: Infinite recursion tests triggering a segfault http://bugs.python.org/issue18075 closed by lukasz.langa #18077: dis.dis raises IndexError http://bugs.python.org/issue18077 closed by serhiy.storchaka #18079: documentation tutorial 3.1.3 typo http://bugs.python.org/issue18079 closed by serhiy.storchaka #18080: setting CC no longer overrides default linker for extension mo http://bugs.python.org/issue18080 closed by ned.deily #18084: wave.py should use sys.byteorder to determine endianess http://bugs.python.org/issue18084 closed by serhiy.storchaka #18086: Create importlib.util.set_attrs_post_import() http://bugs.python.org/issue18086 closed by brett.cannon #18087: os.listdir don't show hidden option http://bugs.python.org/issue18087 closed by serhiy.storchaka #18097: spam http://bugs.python.org/issue18097 closed by amaury.forgeotdarc #18098: "Build Applet.app" build fails on OS X 10.8 http://bugs.python.org/issue18098 closed by ned.deily #18107: 'str(long)' can be made faster http://bugs.python.org/issue18107 closed by arigo #967161: pty.spawn() enhancements http://bugs.python.org/issue967161 closed by akuchling #706406: fix bug #685846: raw_input defers signals http://bugs.python.org/issue706406 closed by amaury.forgeotdarc #592703: HTTPS does not handle pipelined requests http://bugs.python.org/issue592703 closed by amaury.forgeotdarc #1554133: PyOS_InputHook() and related API funcs. not documented http://bugs.python.org/issue1554133 closed by akuchling

1 0

PEP 443 - request for pronouncement
by Łukasz Langa 31 May '13

31 May '13

Hello python-dev, PEP 443 is ready for final review. I'm attaching the latest version below for convenience. The full history of changes is available here: http://hg.python.org/peps/log/tip/pep-0443.txt A reference implementation for PEP 443 is available at: http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l363 with relevant tests here: http://hg.python.org/features/pep-443/file/tip/Lib/test/test_functools.py#l… and documentation here: http://hg.python.org/features/pep-443/file/tip/Doc/library/functools.rst#l1… There's also an official backport for 2.6 - 3.3 already up: https://pypi.python.org/pypi/singledispatch PEP: 443 Title: Single-dispatch generic functions Version: $Revision$ Last-Modified: $Date$ Author: Łukasz Langa <lukasz(a)langa.pl> Discussions-To: Python-Dev <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 22-May-2013 Post-History: 22-May-2013, 25-May-2013, 31-May-2013 Replaces: 245, 246, 3124 Abstract ======== This PEP proposes a new mechanism in the ``functools`` standard library module that provides a simple form of generic programming known as single-dispatch generic functions. A **generic function** is composed of multiple functions implementing the same operation for different types. Which implementation should be used during a call is determined by the dispatch algorithm. When the implementation is chosen based on the type of a single argument, this is known as **single dispatch**. Rationale and Goals =================== Python has always provided a variety of built-in and standard-library generic functions, such as ``len()``, ``iter()``, ``pprint.pprint()``, ``copy.copy()``, and most of the functions in the ``operator`` module. However, it currently: 1. does not have a simple or straightforward way for developers to create new generic functions, 2. does not have a standard way for methods to be added to existing generic functions (i.e., some are added using registration functions, others require defining ``__special__`` methods, possibly by monkeypatching). In addition, it is currently a common anti-pattern for Python code to inspect the types of received arguments, in order to decide what to do with the objects. For example, code may wish to accept either an object of some type, or a sequence of objects of that type. Currently, the "obvious way" to do this is by type inspection, but this is brittle and closed to extension. Abstract Base Classes make it easier to discover present behaviour, but don't help adding new behaviour. A developer using an already-written library may be unable to change how their objects are treated by such code, especially if the objects they are using were created by a third party. Therefore, this PEP proposes a uniform API to address dynamic overloading using decorators. User API ======== To define a generic function, decorate it with the ``@singledispatch`` decorator. Note that the dispatch happens on the type of the first argument, create your function accordingly:: >>> from functools import singledispatch >>> @singledispatch ... def fun(arg, verbose=False): ... if verbose: ... print("Let me just say,", end=" ") ... print(arg) To add overloaded implementations to the function, use the ``register()`` attribute of the generic function. It is a decorator, taking a type parameter and decorating a function implementing the operation for that type:: >>> @fun.register(int) ... def _(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) ... >>> @fun.register(list) ... def _(arg, verbose=False): ... if verbose: ... print("Enumerate this:") ... for i, elem in enumerate(arg): ... print(i, elem) To enable registering lambdas and pre-existing functions, the ``register()`` attribute can be used in a functional form:: >>> def nothing(arg, verbose=False): ... print("Nothing.") ... >>> fun.register(type(None), nothing) The ``register()`` attribute returns the undecorated function which enables decorator stacking, pickling, as well as creating unit tests for each variant independently:: >>> @fun.register(float) ... @fun.register(Decimal) ... def fun_num(arg, verbose=False): ... if verbose: ... print("Half of your number:", end=" ") ... print(arg / 2) ... >>> fun_num is fun False When called, the generic function dispatches on the type of the first argument:: >>> fun("Hello, world.") Hello, world. >>> fun("test.", verbose=True) Let me just say, test. >>> fun(42, verbose=True) Strength in numbers, eh? 42 >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True) Enumerate this: 0 spam 1 spam 2 eggs 3 spam >>> fun(None) Nothing. >>> fun(1.23) 0.615 Where there is no registered implementation for a specific type, its method resolution order is used to find a more generic implementation. To check which implementation will the generic function choose for a given type, use the ``dispatch()`` attribute:: >>> fun.dispatch(float) <function fun_num at 0x104319058> >>> fun.dispatch(dict) <function fun at 0x103fe4788> To access all registered implementations, use the read-only ``registry`` attribute:: >>> fun.registry.keys() dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>, <class 'decimal.Decimal'>, <class 'list'>, <class 'float'>]) >>> fun.registry[float] <function fun_num at 0x1035a2840> >>> fun.registry[object] <function fun at 0x103170788> The proposed API is intentionally limited and opinionated, as to ensure it is easy to explain and use, as well as to maintain consistency with existing members in the ``functools`` module. Implementation Notes ==================== The functionality described in this PEP is already implemented in the ``pkgutil`` standard library module as ``simplegeneric``. Because this implementation is mature, the goal is to move it largely as-is. The reference implementation is available on hg.python.org [#ref-impl]_. The dispatch type is specified as a decorator argument. An alternative form using function annotations has been considered but its inclusion has been deferred. As of May 2013, this usage pattern is out of scope for the standard library [#pep-0008]_ and the best practices for annotation usage are still debated. Based on the current ``pkgutil.simplegeneric`` implementation and following the convention on registering virtual subclasses on Abstract Base Classes, the dispatch registry will not be thread-safe. Abstract Base Classes --------------------- The ``pkgutil.simplegeneric`` implementation relied on several forms of method resultion order (MRO). ``@singledispatch`` removes special handling of old-style classes and Zope's ExtensionClasses. More importantly, it introduces support for Abstract Base Classes (ABC). When a generic function implementation is registered for an ABC, the dispatch algorithm switches to a mode of MRO calculation for the provided argument which includes the relevant ABCs. The algorithm is as follows:: def _compose_mro(cls, haystack): """Calculates the MRO for a given class `cls`, including relevant abstract base classes from `haystack`.""" bases = set(cls.__mro__) mro = list(cls.__mro__) for regcls in haystack: if regcls in bases or not issubclass(cls, regcls): continue # either present in the __mro__ or unrelated for index, base in enumerate(mro): if not issubclass(base, regcls): break if base in bases and not issubclass(regcls, base): # Conflict resolution: put classes present in __mro__ # and their subclasses first. index += 1 mro.insert(index, regcls) return mro In its most basic form, it returns the MRO for the given type:: >>> _compose_mro(dict, []) [<class 'dict'>, <class 'object'>] When the haystack consists of ABCs that the specified type is a subclass of, they are inserted in a predictable order:: >>> _compose_mro(dict, [Sized, MutableMapping, str, ... Sequence, Iterable]) [<class 'dict'>, <class 'collections.abc.MutableMapping'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Sized'>, <class 'object'>] While this mode of operation is significantly slower, all dispatch decisions are cached. The cache is invalidated on registering new implementations on the generic function or when user code calls ``register()`` on an ABC to register a new virtual subclass. In the latter case, it is possible to create a situation with ambiguous dispatch, for instance:: >>> from collections import Iterable, Container >>> class P: ... pass >>> Iterable.register(P) <class '__main__.P'> >>> Container.register(P) <class '__main__.P'> Faced with ambiguity, ``@singledispatch`` refuses the temptation to guess:: >>> @singledispatch ... def g(arg): ... return "base" ... >>> g.register(Iterable, lambda arg: "iterable") <function <lambda> at 0x108b49110> >>> g.register(Container, lambda arg: "container") <function <lambda> at 0x108b491c8> >>> g(P()) Traceback (most recent call last): ... RuntimeError: Ambiguous dispatch: <class 'collections.abc.Container'> or <class 'collections.abc.Iterable'> Note that this exception would not be raised if ``Iterable`` and ``Container`` had been provided as base classes during class definition. In this case dispatch happens in the MRO order:: >>> class Ten(Iterable, Container): ... def __iter__(self): ... for i in range(10): ... yield i ... def __contains__(self, value): ... return value in range(10) ... >>> g(Ten()) 'iterable' Usage Patterns ============== This PEP proposes extending behaviour only of functions specifically marked as generic. Just as a base class method may be overridden by a subclass, so too may a function be overloaded to provide custom functionality for a given type. Universal overloading does not equal *arbitrary* overloading, in the sense that we need not expect people to randomly redefine the behavior of existing functions in unpredictable ways. To the contrary, generic function usage in actual programs tends to follow very predictable patterns and registered implementations are highly-discoverable in the common case. If a module is defining a new generic operation, it will usually also define any required implementations for existing types in the same place. Likewise, if a module is defining a new type, then it will usually define implementations there for any generic functions that it knows or cares about. As a result, the vast majority of registered implementations can be found adjacent to either the function being overloaded, or to a newly-defined type for which the implementation is adding support. It is only in rather infrequent cases that one will have implementations registered in a module that contains neither the function nor the type(s) for which the implementation is added. In the absence of incompetence or deliberate intention to be obscure, the few implementations that are not registered adjacent to the relevant type(s) or function(s), will generally not need to be understood or known about outside the scope where those implementations are defined. (Except in the "support modules" case, where best practice suggests naming them accordingly.) As mentioned earlier, single-dispatch generics are already prolific throughout the standard library. A clean, standard way of doing them provides a way forward to refactor those custom implementations to use a common one, opening them up for user extensibility at the same time. Alternative approaches ====================== In PEP 3124 [#pep-3124]_ Phillip J. Eby proposes a full-grown solution with overloading based on arbitrary rule sets (with the default implementation dispatching on argument types), as well as interfaces, adaptation and method combining. PEAK-Rules [#peak-rules]_ is a reference implementation of the concepts described in PJE's PEP. Such a broad approach is inherently complex, which makes reaching a consensus hard. In contrast, this PEP focuses on a single piece of functionality that is simple to reason about. It's important to note this does not preclude the use of other approaches now or in the future. In a 2005 article on Artima [#artima2005]_ Guido van Rossum presents a generic function implementation that dispatches on types of all arguments on a function. The same approach was chosen in Andrey Popp's ``generic`` package available on PyPI [#pypi-generic]_, as well as David Mertz's ``gnosis.magic.multimethods`` [#gnosis-multimethods]_. While this seems desirable at first, I agree with Fredrik Lundh's comment that "if you design APIs with pages of logic just to sort out what code a function should execute, you should probably hand over the API design to someone else". In other words, the single argument approach proposed in this PEP is not only easier to implement but also clearly communicates that dispatching on a more complex state is an anti-pattern. It also has the virtue of corresponding directly with the familiar method dispatch mechanism in object oriented programming. The only difference is whether the custom implementation is associated more closely with the data (object-oriented methods) or the algorithm (single-dispatch overloading). PyPy's RPython offers ``extendabletype`` [#pairtype]_, a metaclass which enables classes to be externally extended. In combination with ``pairtype()`` and ``pair()`` factories, this offers a form of single-dispatch generics. Acknowledgements ================ Apart from Phillip J. Eby's work on PEP 3124 [#pep-3124]_ and PEAK-Rules, influences include Paul Moore's original issue [#issue-5135]_ that proposed exposing ``pkgutil.simplegeneric`` as part of the ``functools`` API, Guido van Rossum's article on multimethods [#artima2005]_, and discussions with Raymond Hettinger on a general pprint rewrite. Huge thanks to Nick Coghlan for encouraging me to create this PEP and providing initial feedback. References ========== .. [#ref-impl] http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l359 .. [#pep-0008] PEP 8 states in the "Programming Recommendations" section that "the Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style". (http://www.python.org/dev/peps/pep-0008) .. [#pep-3124] http://www.python.org/dev/peps/pep-3124/ .. [#peak-rules] http://peak.telecommunity.com/DevCenter/PEAK_2dRules .. [#artima2005] http://www.artima.com/weblogs/viewpost.jsp?thread=101605 .. [#pypi-generic] http://pypi.python.org/pypi/generic .. [#gnosis-multimethods] http://gnosis.cx/publish/programming/charming_python_b12.html .. [#pairtype] https://bitbucket.org/pypy/pypy/raw/default/rpython/tool/pairtype.py .. [#issue-5135] http://bugs.python.org/issue5135 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

2 3

PEP 443 - Single-dispatch generic functions (including ABC support)
by Łukasz Langa 31 May '13

31 May '13

Hello, Since the initial version, several minor changes have been made to the PEP. The history is visible on hg.python.org. The most important change in this version is that I introduced ABC support and completed a reference implementation. No open issues remain from my point of view. PEP: 443 Title: Single-dispatch generic functions Version: $Revision$ Last-Modified: $Date$ Author: Łukasz Langa <lukasz(a)langa.pl> Discussions-To: Python-Dev <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 22-May-2013 Post-History: 22-May-2013, 25-May-2013 Replaces: 245, 246, 3124 Abstract ======== This PEP proposes a new mechanism in the ``functools`` standard library module that provides a simple form of generic programming known as single-dispatch generic functions. A **generic function** is composed of multiple functions sharing the same name. Which form should be used during a call is determined by the dispatch algorithm. When the implementation is chosen based on the type of a single argument, this is known as **single dispatch**. Rationale and Goals =================== Python has always provided a variety of built-in and standard-library generic functions, such as ``len()``, ``iter()``, ``pprint.pprint()``, ``copy.copy()``, and most of the functions in the ``operator`` module. However, it currently: 1. does not have a simple or straightforward way for developers to create new generic functions, 2. does not have a standard way for methods to be added to existing generic functions (i.e., some are added using registration functions, others require defining ``__special__`` methods, possibly by monkeypatching). In addition, it is currently a common anti-pattern for Python code to inspect the types of received arguments, in order to decide what to do with the objects. For example, code may wish to accept either an object of some type, or a sequence of objects of that type. Currently, the "obvious way" to do this is by type inspection, but this is brittle and closed to extension. Abstract Base Classes make it easier to discover present behaviour, but don't help adding new behaviour. A developer using an already-written library may be unable to change how their objects are treated by such code, especially if the objects they are using were created by a third party. Therefore, this PEP proposes a uniform API to address dynamic overloading using decorators. User API ======== To define a generic function, decorate it with the ``@singledispatch`` decorator. Note that the dispatch happens on the type of the first argument, create your function accordingly:: >>> from functools import singledispatch >>> @singledispatch ... def fun(arg, verbose=False): ... if verbose: ... print("Let me just say,", end=" ") ... print(arg) To add overloaded implementations to the function, use the ``register()`` attribute of the generic function. It takes a type parameter:: >>> @fun.register(int) ... def _(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) ... >>> @fun.register(list) ... def _(arg, verbose=False): ... if verbose: ... print("Enumerate this:") ... for i, elem in enumerate(arg): ... print(i, elem) To enable registering lambdas and pre-existing functions, the ``register()`` attribute can be used in a functional form:: >>> def nothing(arg, verbose=False): ... print("Nothing.") ... >>> fun.register(type(None), nothing) The ``register()`` attribute returns the undecorated function which enables decorator stacking, pickling, as well as creating unit tests for each variant independently:: >>> @fun.register(float) ... @fun.register(Decimal) ... def fun_num(arg, verbose=False): ... if verbose: ... print("Half of your number:", end=" ") ... print(arg / 2) ... >>> fun_num is fun False When called, the generic function dispatches on the first argument:: >>> fun("Hello, world.") Hello, world. >>> fun("test.", verbose=True) Let me just say, test. >>> fun(42, verbose=True) Strength in numbers, eh? 42 >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True) Enumerate this: 0 spam 1 spam 2 eggs 3 spam >>> fun(None) Nothing. >>> fun(1.23) 0.615 To get the implementation for a specific type, use the ``dispatch()`` attribute:: >>> fun.dispatch(float) <function fun_num at 0x104319058> >>> fun.dispatch(dict) <function fun at 0x103fe4788> The proposed API is intentionally limited and opinionated, as to ensure it is easy to explain and use, as well as to maintain consistency with existing members in the ``functools`` module. Implementation Notes ==================== The functionality described in this PEP is already implemented in the ``pkgutil`` standard library module as ``simplegeneric``. Because this implementation is mature, the goal is to move it largely as-is. The reference implementation is available on hg.python.org [#ref-impl]_. The dispatch type is specified as a decorator argument. An alternative form using function annotations has been considered but its inclusion has been deferred. As of May 2013, this usage pattern is out of scope for the standard library [#pep-0008]_ and the best practices for annotation usage are still debated. Based on the current ``pkgutil.simplegeneric`` implementation and following the convention on registering virtual subclasses on Abstract Base Classes, the dispatch registry will not be thread-safe. Abstract Base Classes --------------------- The ``pkgutil.simplegeneric`` implementation relied on several forms of method resultion order (MRO). ``@singledispatch`` removes special handling of old-style classes and Zope's ExtensionClasses. More importantly, it introduces support for Abstract Base Classes (ABC). When a generic function overload is registered for an ABC, the dispatch algorithm switches to a mode of MRO calculation for the provided argument which includes the relevant ABCs. The algorithm is as follows:: def _compose_mro(cls, haystack): """Calculates the MRO for a given class `cls`, including relevant abstract base classes from `haystack`.""" bases = set(cls.__mro__) mro = list(cls.__mro__) for regcls in haystack: if regcls in bases or not issubclass(cls, regcls): continue # either present in the __mro__ or unrelated for index, base in enumerate(mro): if not issubclass(base, regcls): break if base in bases and not issubclass(regcls, base): # Conflict resolution: put classes present in __mro__ # and their subclasses first. index += 1 mro.insert(index, regcls) return mro While this mode of operation is significantly slower, no caching is involved because user code may ``register()`` a new class on an ABC at any time. In such case, it is possible to create a situation with ambiguous dispatch, for instance:: >>> from collections import Iterable, Container >>> class P: ... pass >>> Iterable.register(P) <class '__main__.P'> >>> Container.register(P) <class '__main__.P'> Faced with ambiguity, ``@singledispatch`` refuses the temptation to guess:: >>> @singledispatch ... def g(arg): ... return "base" ... >>> g.register(Iterable, lambda arg: "iterable") <function <lambda> at 0x108b49110> >>> g.register(Container, lambda arg: "container") <function <lambda> at 0x108b491c8> >>> g(P()) Traceback (most recent call last): ... RuntimeError: Ambiguous dispatch: <class 'collections.abc.Container'> or <class 'collections.abc.Iterable'> Note that this exception would not be raised if ``Iterable`` and ``Container`` had been provided as base classes during class definition. In this case dispatch happens in the MRO order:: >>> class Ten(Iterable, Container): ... def __iter__(self): ... for i in range(10): ... yield i ... def __contains__(self, value): ... return value in range(10) ... >>> g(Ten()) 'iterable' Usage Patterns ============== This PEP proposes extending behaviour only of functions specifically marked as generic. Just as a base class method may be overridden by a subclass, so too may a function be overloaded to provide custom functionality for a given type. Universal overloading does not equal *arbitrary* overloading, in the sense that we need not expect people to randomly redefine the behavior of existing functions in unpredictable ways. To the contrary, generic function usage in actual programs tends to follow very predictable patterns and overloads are highly-discoverable in the common case. If a module is defining a new generic operation, it will usually also define any required overloads for existing types in the same place. Likewise, if a module is defining a new type, then it will usually define overloads there for any generic functions that it knows or cares about. As a result, the vast majority of overloads can be found adjacent to either the function being overloaded, or to a newly-defined type for which the overload is adding support. It is only in rather infrequent cases that one will have overloads in a module that contains neither the function nor the type(s) for which the overload is added. In the absence of incompetence or deliberate intention to be obscure, the few overloads that are not adjacent to the relevant type(s) or function(s), will generally not need to be understood or known about outside the scope where those overloads are defined. (Except in the "support modules" case, where best practice suggests naming them accordingly.) As mentioned earlier, single-dispatch generics are already prolific throughout the standard library. A clean, standard way of doing them provides a way forward to refactor those custom implementations to use a common one, opening them up for user extensibility at the same time. Alternative approaches ====================== In PEP 3124 [#pep-3124]_ Phillip J. Eby proposes a full-grown solution with overloading based on arbitrary rule sets (with the default implementation dispatching on argument types), as well as interfaces, adaptation and method combining. PEAK-Rules [#peak-rules]_ is a reference implementation of the concepts described in PJE's PEP. Such a broad approach is inherently complex, which makes reaching a consensus hard. In contrast, this PEP focuses on a single piece of functionality that is simple to reason about. It's important to note this does not preclude the use of other approaches now or in the future. In a 2005 article on Artima [#artima2005]_ Guido van Rossum presents a generic function implementation that dispatches on types of all arguments on a function. The same approach was chosen in Andrey Popp's ``generic`` package available on PyPI [#pypi-generic]_, as well as David Mertz's ``gnosis.magic.multimethods`` [#gnosis-multimethods]_. While this seems desirable at first, I agree with Fredrik Lundh's comment that "if you design APIs with pages of logic just to sort out what code a function should execute, you should probably hand over the API design to someone else". In other words, the single argument approach proposed in this PEP is not only easier to implement but also clearly communicates that dispatching on a more complex state is an anti-pattern. It also has the virtue of corresponding directly with the familiar method dispatch mechanism in object oriented programming. The only difference is whether the custom implementation is associated more closely with the data (object-oriented methods) or the algorithm (single-dispatch overloading). PyPy's RPython offers ``extendabletype`` [#pairtype]_, a metaclass which enables classes to be externally extended. In combination with ``pairtype()`` and ``pair()`` factories, this offers a form of single-dispatch generics. Acknowledgements ================ Apart from Phillip J. Eby's work on PEP 3124 [#pep-3124]_ and PEAK-Rules, influences include Paul Moore's original issue [#issue-5135]_ that proposed exposing ``pkgutil.simplegeneric`` as part of the ``functools`` API, Guido van Rossum's article on multimethods [#artima2005]_, and discussions with Raymond Hettinger on a general pprint rewrite. Huge thanks to Nick Coghlan for encouraging me to create this PEP and providing initial feedback. References ========== .. [#ref-impl] http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l359 .. [#pep-0008] PEP 8 states in the "Programming Recommendations" section that "the Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style". (http://www.python.org/dev/peps/pep-0008) .. [#pep-3124] http://www.python.org/dev/peps/pep-3124/ .. [#peak-rules] http://peak.telecommunity.com/DevCenter/PEAK_2dRules .. [#artima2005] http://www.artima.com/weblogs/viewpost.jsp?thread=101605 .. [#pypi-generic] http://pypi.python.org/pypi/generic .. [#gnosis-multimethods] http://gnosis.cx/publish/programming/charming_python_b12.html .. [#pairtype] https://bitbucket.org/pypy/pypy/raw/default/rpython/tool/pairtype.py .. [#issue-5135] http://bugs.python.org/issue5135 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

7 28

performance testing recommendations in devguide
by Eric Snow 30 May '13

30 May '13

The devguide doesn't have anything on performance testing that I could find. We do have a number of relatively useful resources in this space though, like pybench and (eventually) speed.python.org. I'd like to add a page to the devguide on performance testing, including an explanation of our performance goals, how to test for them, and what tools are available. Tools I'm aware of: * pybench (relatively limited in real-world usefulness) * timeit module (for quick comparisions) * benchmarks repo (real-world performance test suite) * speed.python.org (would omit for now) Things to test: * speed * memory (tools? tests?) Critically sensitive performance subjects * interpreter start-up time * module import overhead * attribute lookup overhead (including MRO traversal) * function call overhead * instance creation overhead * dict performance (the underlying namespace type) * tuple performance (packing/unpacking, integral container type) * string performance What would be important to say in the devguide regarding Python performance and testing it? What would you add/subtract from the above? How important is testing memory performance? How do we avoid performance regressions? Thanks! -eric

7 7

Segmentation fault on 3.4 with --pydebug
by Łukasz Langa 30 May '13

30 May '13

This happens after Benjamin's changes in 83937. Anybody else seeing this? Intel i5 2.4 GHz, Mac OS X 10.8.3, clang $ hg up default $ make distclean $ MACOSX_DEPLOYMENT_TARGET=10.8 ./configure --with-pydebug $ make $ ./python.exe -Wd -m test.regrtest test_exceptions [1/1] test_exceptions Fatal Python error: Segmentation fault Current thread 0x00007fff74254180: File "/Users/ambv/Documents/Projekty/Python/cpython/py34/Lib/test/test_exceptions.py", line 453 in f File "/Users/ambv/Documents/Projekty/Python/cpython/py34/Lib/test/test_exceptions.py", line 453 in f File "/Users/ambv/Documents/Projekty/Python/cpython/py34/Lib/test/test_exceptions.py", line 453 in f ... (repeated a 100 times) Command terminated abnormally. Everything runs fine without --with-pydebug (or before 83937 with --with-pydebug). -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

5 5