Mailman 3 July 2013 - Python-Dev

Re: [Python-Dev] [Python-checkins] cpython (merge default -> 3.3): merge 3.3 (#18470)
by Victor Stinner July 16, 2013

July 16, 2013

2013/7/16 benjamin.peterson <python-checkins(a)python.org>: > http://hg.python.org/cpython/rev/8889c9b5dd3a > changeset: 84654:8889c9b5dd3a > branch: 3.3 > parent: 84653:c3a510b22218 > parent: 84652:8a078bf3cf14 > user: Benjamin Peterson <benjamin(a)python.org> > date: Mon Jul 15 19:15:49 2013 -0700 > summary: > merge 3.3 (#18470) This commit merges branches default and 3.3 into 3.3. Usually, we merge branches 3.3 and … [View More]

1 0

PEP 447: hooking into super() attribute resolution
by Ronald Oussoren July 15, 2013

July 15, 2013

Hi, I've posted a new update of my proposal to add a way to override the attribute resolution proces by super(). I've rewritten the PEP and implementation based on feedback by Steve Dower. In the current edition of the proposal the hook is a method on the type, defined in a metaclass for types. A simple demo: class meta (type): def __locallookup__(cls, name): return type.__locallookup__(cls, name.upper()) class S (metaclass=meta): def m(self): … [View More]return 42 def M(self): return "fortytwo" class SS (S): def m(self): return 21 def M(self): return "twentyone" o = SS() print("self", o.m()) print("super", super(SS, o).m()) The last line currently prints "super 42" and would print "super fortytwo" with my proposal. A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details). I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex. Ronald PEP: 447 Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren(a)mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, ? Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. This new hook makes it possible to affect attribute lookup for both normal attribute lookup and lookup through the `super class`_. The superclass attribute lookup hook ==================================== Both ``super.__getattribute__`` and ``object.__getattribute__`` (or `PyObject_GenericGetAttr`_ in C code) walk an object's MRO and peek in the class' ``__dict__`` to look up attributes. A way to affect this lookup is using a method on the meta class for the type, that by default looks up the name in the class ``__dict__``. In Python code -------------- A meta type can define a method ``__locallookup__`` that is called during attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``:: class MetaType (type): def __locallookup__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None The example method above is pseudo code for the implementation of this method on `type`_ (the actual implementation is in C, and doesn't suffer from the recursion problem in this example). The ``__locallookup__`` method has as its arguments a class and the name of the attribute that is looked up. It should return the value of the attribute without invoking descriptors, or raise `AttributeError`_ when the name cannot be found. In C code --------- A new slot ``tp_locallookup`` is added to the ``PyTypeObject`` struct, this slot corresponds to the ``__locallookup__`` method on `type`_. The slot has the following prototype:: PyObject* (*locallookupfunc)(PyTypeObject* cls, PyObject* name); This method should lookup *name* in the namespace of *cls*, without looking at superclasses, and should not invoke descriptors. The method returns ``NULL`` without setting an exception when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference). Usage of this hook ------------------ The new method will be defined for `type`_, and will peek in the class dictionary:: static PyObject* type_locallookup(PyTypeObject* cls, PyObject* name) { PyObject* res; if (!cls->tp_dict) { return NULL; } res = PyDict_GetItem(cls, name); Py_XINCREF(res); return res; } The new method will be used by both `PyObject_GenericGetAttr`_ and ``super.__getattribute__`` instead of peeking in a type's ``tp_dict``. Alternative proposals --------------------- ``__getattribute_super__`` .......................... An earlier version of this PEP used the following static method on classes:: def __getattribute_super__(cls, name, object, owner): pass This method performed name lookup as well as invoking descriptors and was necessarily limited to working only with ``super.__getattribute__``. Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. That won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * Should the python method raise an exception or return a magic value (such as the `NotImplemented`_ return value used by comparison operators). The latter could be slightly faster because it doesn't have to overhead of setting up exception state, but makes it impossible to use that value as an attribute on a class. * The proposed change to `PyObject_GenericGetAttr`_ will probably cause problems with the attribute lookup cache (MCACHE): 1. That code stores borrowed references, which won't work when the hook is present. That is mostly fixable, but at the cost of possibly keeping garbage alive. 2. Caching isn't an option when a hook might execute arbitrary code (and there hence is no reason to assume that the hooks return value won't change later on). The only workaround I could find for this is to make the hook a fallback (that is, more like ``__getattr__`` than ``__getattribute__``). References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html#super .. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented .. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr .. _`type`: http://docs.python.org/3/library/functions.html#type .. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError [View Less]

2 3

Hooking into super() attribute resolution
by Ronald Oussoren July 15, 2013

July 15, 2013

Hi, Below is a very preliminary draft PEP for adding a special method that can be used to hook into the attribute resolution process of the super object. The primary usecase for using this special method are classes that perform custom logic in their __getattribute__ method, where the default behavior of super (peekin the the class __dict__) is not appropriate. The primary reason I wrote this proposal is PyObjC: it dynamicly looks up methods in its __getattribute__ and caches the result in … [View More]the class __dict__, because of this super() will often not work correctly and therefore I'm currently shipping a custom subclass of super() that basicly contains an in-line implementation of the hook that would be used by PyObjC. I have a partial implementation of the hook system in issue 18181 and a PyObjC patch that uses it. The implementation currently does not contain tests, and I'm sure that I'll find edge cases that I haven't thought about yet when I add tests. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren(a)mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013 Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook ==================================== In C code --------- A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, otherwise it will peek in the class ``tp_dict``. The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* tp, PyObject* self, PyObject* name); The function should perform attribute lookup for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. In Python code -------------- A Python class can contain a definition for a method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(self, cls, name): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* Alternative proposals --------------------- Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? References ========== * `Issue 18181`_ contains a prototype implementation The prototype uses different names than this proposal. Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super [View Less]

3 9

PLY in stdlib (was cffi in stdlib)
by David Beazley July 15, 2013

July 15, 2013

> > From: Eli Bendersky <eliben(a)gmail.com> > > I'll be the first one to admit that pycparser is almost certainly not > generally useful enough to be exposed in the stdlib. So just using it as an > implementation detail is absolutely fine. PLY is a more interesting > question, however, since PLY is somewhat more generally useful. That said, > I see all this as implementation details that shouldn't distract us from > the main point of whether cffi should be … [View More]

9 13

Doubt about closure/function generation in bytecode
by Andrea Griffini July 15, 2013

July 15, 2013

Is there any semantic difference between BUILD_TUPLE 0 LOAD_CONST <code object ... > MAKE_CLOSURE 0 and LOAD_CONST <code object ...> MAKE_FUNCTION 0 ? In other words is there any difference between a function and a closure that doesn't capture anything? Is it just a speed optimization for a common case? Andrea Griffini

2 1

Why is the return value of __contains__ coerced to boolean, but that of __lt__ and the like is not?
by Ben Hoyt July 15, 2013

July 15, 2013

I'm curious why the return value of __contains__ is coerced to True or False, whereas the return value of "normal" comparison operators like __lt__ and the like are not. The latter return the value directly without forcing it to be True or False. This makes overriding __contains__ significantly less flexible, so I'm wondering why it's designed or implemented that way. (I believe it's the cmp_outcome() function in Python/ceval.c that does this: http://hg.python.org/cpython/file/db9fe49069ed/… [View More]

3 4

Summary of Python tracker Issues
by Python tracker July 12, 2013

July 12, 2013

ACTIVITY SUMMARY (2013-07-05 - 2013-07-12) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4067 ( -5) closed 26140 (+74) total 30207 (+69) Open issues with patches: 1834 Issues opened (49) ================== #7457: Adding a read_pkg_file to DistributionMetadata http://bugs.python.org/issue7457 reopened by jason.coombs #18366: Look into using … [View More]

1 0

Accepting PEP 445
by Antoine Pitrou July 9, 2013

July 9, 2013

Hello, I'm accepting PEP 445 (A C API to customize memory allocators) by Victor. There is probably some grammar to correct here and there (neither Victor nor I are native English speakers), but I don't want this to hold back acceptance. The PEP is an obvious improvement in flexibility for everyone embedding Python in some external application, or even recompiling their own Python runtime for specific uses. The implementation is tracked in http://bugs.python.org/issue3329 Thanks to Victor … [View More]

5 7

pyfailmalloc: new debug tool to test how your applications handles MemoryError
by Victor Stinner July 8, 2013

July 8, 2013

Hi, The PEP 445 (Add new APIs to customize Python memory allocators) has been accepted, I commited its implementation. So it's time to have fun with this API. I developed a small Python module (150 lines of C code) to inject memory allocation failures: https://pypi.python.org/pypi/pyfailmalloc This module can be used to test your application under very low memory condition, test how your application handles MemoryError. I ran it in the Python test suite: Python core and stdlib do not handle … [View More]

1 1

Rough idea for adding introspection information for builtins
by Larry Hastings July 8, 2013

July 8, 2013

The original impetus for Argument Clinic was adding introspection information for builtins--it seemed like any manual approach I came up with would push the builtins maintenance burden beyond the pale. Assuming that we have Argument Clinic or something like it, we don't need to optimize for ease of use from the API end--we can optimize for data size. So the approach writ large: store a blob of data associated with each entry point, as small as possible. Reconstitute the appropriate … [View More]inspect.Signature on demand by reading that blob. Where to store the data? PyMethodDef is the obvious spot, but I think that structure is part of the stable ABI. So we'd need a new PyMethodDefEx and that'd be a little tiresome. Less violent to the ABI would be defining a new array of pointers-to-introspection-blobs, parallel to the PyMethodDef array, passed in via a new entry point. On to the representation. Consider the function def foo(arg, b=3, *, kwonly='a'): pass I considered four approaches, each listed below along with its total size if it was stored as C static data. 1. A specialized bytecode format, something like pickle, like this: bytes([ PARAMETER_START_LENGTH_3, 'a', 'r', 'g', PARAMETER_START_LENGTH_1, 'b', PARAMETER_DEFAULT_LENGTH_1, '3', KEYWORD_ONLY, PARAMETER_START_LENGTH_6, 'k', 'w', 'o', 'n', 'l', 'y', PARAMETER_DEFAULT_LENGTH_3, '\'', 'a', '\'', END ]) Length: 20 bytes. 2. Just use pickle--pickle the result of inspect.signature() run on a mocked-up signature, just store that. Length: 130 bytes. (Assume a two-byte size stored next to it.) 3. Store a string that, if eval'd, would produce the inspect.Signature. Length: 231 bytes. (This could be made smaller if we could assume "from inspect import *" or "p = inspect.Parameter" or something, but it'd still be easily the heaviest.) 4. Store a string that looks like the Python declaration of the signature, and parse it (Nick's suggestion). For foo above, this would be "(arg,b=3,*,kwonly='a')". Length: 23 bytes. Of those, Nick's suggestion seems best. It's slightly bigger than the specialized bytecode format, but it's human-readable (and human-writable!), and it'd be the easiest to implement. My first idea for implementation: add a "def x" to the front and ": pass" to the end, then run it through ast.parse. Iterate over the tree, converting parameters into inspect.Parameters and handling the return annotation if present. Default values and annotations would be turned into values by ast.eval_literal. (It wouldn't surprise me if there's a cleaner way to do it than the fake function definition; I'm not familiar with the ast module.) We'd want one more mild hack: the DSL will support positional parameters, and inspect.Signature supports positional parameters, so it'd be nice to render that information. But we can't represent that in Python syntax (or at least not yet!), so we can't let ast.parse see it. My suggestion: run it through ast.parse, and if it throws a SyntaxError see if the problem was a slash. If it was, remove the slash, reprocess through ast.parse, and remember that all parameters are positional-only (and barf if there are kwonly, args, or kwargs). Thoughts? //arry/ [View Less]

9 23