Mailman 3 February 2013 - Python-Dev

Status of the PEP 433?
by Victor Stinner Feb. 14, 2013

Feb. 14, 2013

Hi, I read again all emails related to PEP 433. There are different opposition to this change and I tried to make a summary. My question is: how can I move this PEP forward (accept, defer or reject it)? Can I accept it myself, or do we need a global agreement, or should someone else decide for this PEP (who? Antoine maybe)? -- Most complains concern issues if cloexec is set by default. I don't understand why so many people feel concerned because my PEP proposes *to not change anything to the default behaviour*: cloexec will not be set by default, as it was always the case since Python 1. You have to explicitly use -e command line option, PYTHONCLOEXEC=1 environment variable, or call sys.setdefaultcloexec(True). If you *explicitly* change the default value of the cloexec parameter, you must expect to change the behaviour of your application and maybe discover "bugs" in your application (application incompatible with cloexec set by default). (I don't think that any library would like to play with sys.setdefaultcloexec(), since most developers agree that it might break applications.) -- Ok, let's see all oppositions, presented as a FAQ. (I hope that I didn't misuse email quotes.) Q: the PEP does not solve file descriptor inherance after fork() A: Correct, it only solves issues with exec(), as explained in the PEP. Q: the PEP changes the POSIX behaviour. A: Antoine replied that "Python is not POSIX", Nick wrote "While many of Python's APIs are heavily inspired by POSIX, it still isn't POSIX" (he was talking about integer division). Perl sets cloexec flag by default since its first version, something like 25 years ago, except for explicit POSIX::* calls. Ruby 2 will also set cloexec flag by default. Q: The PEP may break applications. A: Most developers agree that it is (very) unlikely. If file descriptor inherance matters, subprocess must be used (because it rocks!) with pass_fds. If subprocess is used without pass_fds, applications stops working since python 3.2 (since python 3.0 on Windows). pass_fds will clear the close-on-exec flag on the file descriptors with the PEP. If an application breaks because it relies on file descriptor inherance, it's easy to detect it (EBADF error) and simple to fix it (ex: just call sys.setdefaultcloexec(False) at startup, or set explicitly cloexec parameter on the few functions causing trouble). Q: Performance overhead if cloexec is set by default (extra syscalls). A: The PEP now contains a benchmark: the overhead *on a microbenchmark* is between 1 and 3%. I bet that nobody will be able to notice a difference on a real application. Q: The default value of cloexec must not be configurable: "libraries cannot rely on it and have to make file descriptors cloexec on an individual basis, since the default flag can be disabled". A: IMO inherance of file descriptor doesn't matter in most cases, so only a few function must be modified to explicitly set or clear cloexec flag. Said differently: libraries should not care of the default value of the cloexec parameter, which should only be configured by applications (not libraries). Q: The default value of cloexec must not be configurable: "just by looking at this code, you have no way to know whether the file descriptor will be inherited by the child process." A: Guido wrote "Honestly, what happened to the idea that we're all adults here? The likelihood that someone is clueless enough to misuse the flag and yet clueful enough to find out how to change it seems remote -- and they pretty much deserve what they get." -- Another alternative (not listed in the PEP yet) is to leave the posix module (its name depends on the platform: posix, nt or ce) unchanged, and only modify the os module to add cloexec parameter. I implemented this alternative in the "posixmod" branch of the PEP 433 repository. It gives a better traceback on error because the cloexec flag is set in Python, so it's possible to know if the error comes from the creation of the file descriptor or setting the flag in the traceback. (Does it really matter? I never seen an error on setting or clearing the flag.) The problem is that it only concernes posix and os modules: the PEP does also modify socket, io and select modules, which don't have "low" and "high" level modules. So it's possible possible to create a socket using the POSIX behaviour. (Does it matter, it was said that Python is not POSIX.) I'm not convinced that we should do something special for posix/os, I prefer the actual version of the PEP (modify also posix). Victor

2 1

Question regarding: Lib/_markupbase.py
by Developer Developer Feb. 14, 2013

Feb. 14, 2013

Hello all, I was having a look at the file: Lib/_markupbase.py (@ 82151), function: "_parse_doctype_element" and have seen something that has caught my attention: if '>' in rawdata[j:]: return rawdata.find(">", j) + 1 Wouldn't it be better to do the following? pos = rawdata.find(">", j) if pos != -1: return pos + 1 Otherwise I think we are scanning rawdata[j:] twice. Best regards

6 8

efficient string concatenation (yep, from 2004)
by Christian Tismer Feb. 14, 2013

Feb. 14, 2013

Hi friends, _efficient string concatenation_ has been a topic in 2004. Armin Rigo proposed a patch with the name of the subject, more precisely: /[Patches] [ python-Patches-980695 ] efficient string concatenation// //on sourceforge.net, on 2004-06-28.// / This patch was finally added to Python 2.4 on 2004-11-30. Some people might remember the larger discussion if such a patch should be accepted at all, because it changes the programming style for many of us from "don't do that, stupid" to "well, you may do it in CPython", which has quite some impact on other implementations (is it fast on Jython, now?). It changed for instance my programming and teaching style a lot, of course! But I think nobody but people heavily involved in PyPy expected this: Now, more than eight years after that patch appeared and made it into 2.4, PyPy (!) still does _not_ have it! Obviously I was mislead by other optimizations, and the fact that this patch was from a/the major author of PyPy who invented the initial patch for CPython. That this would be in PyPy as well sooner or later was without question for me. Wrong... ;-) Yes, I agree that for PyPy it is much harder to implement without the refcounting trick, and probably even more difficult in case of the JIT. But nevertheless, I tried to find any reference to this missing crucial optimization, with no success after an hour (*). And I guess many other people are stepping in the same trap. So I can imagine that PyPy looses some of its speed in many programs, because Armin's great hack did not make it into PyPy, and this is not loudly declared somewhere. I believe the efficiency of string concatenation is something that people assume by default and add it to the vague CPython compatibility claim, if not explicitly told otherwise. ---- Some silly proof, using python 2.7.3 vs PyPy 1.9: > $ cat strconc.py > #!env python > > from timeit import default_timer as timer > > tim = timer() > > s = '' > for i in xrange(100000): > s += 'X' > > tim = timer() - tim > > print 'time for {} concats = {:0.3f}'.format(len(s), tim) > $ python strconc.py > time for 100000 concats = 0.028 > $ pypy strconc.py > time for 100000 concats = 0.804 Something is needed - a patch for PyPy or for the documentation I guess. This is not just some unoptimized function in some module, but it is used all over the place and became a very common pattern since introduced. /How ironic that a foreseen problem occurs _now_, and _there_ :-)// / cheers -- chris (*) http://pypy.readthedocs.org/en/latest/cpython_differences.html http://pypy.org/compat.html http://pypy.org/performance.html -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

8 14

Marking GC details as CPython-only
by Maciej Fijalkowski Feb. 14, 2013

Feb. 14, 2013

Hi I've tried (and failed) to find what GC details (especially finalizer semantics) are CPython only and which ones are not. The best I could find was the documentation of __del__ here: http://docs.python.org/2/reference/datamodel.html Things were pypy differs: * finalizers in pypy will be called only once, even if the object is resurrected. I'm not sure if this is detail or we're just plain incompatible. * pypy breaks cycles and runs finalizers in random order (but topologically correct), hence gc.garbage is always empty. I *think* this part is really just an implementation detail * we're discussing right now about running multiple finalizers. We want to run them in order, but if there is a link a -> b and a becomes unreachable, we want to reserve the right to call finalizer a then finalizer b, even if a.__del__ resurrects a. What do you think? Overall, the __del__ is baaad. Cheers, fijal

6 9

Usage of += on strings in loops in stdlib
by Maciej Fijalkowski Feb. 14, 2013

Feb. 14, 2013

Hi We recently encountered a performance issue in stdlib for pypy. It turned out that someone commited a performance "fix" that uses += for strings instead of "".join() that was there before. Now this hurts pypy (we can mitigate it to some degree though) and possible Jython and IronPython too. How people feel about generally not having += on long strings in stdlib (since the refcount = 1 thing is a hack)? What about other performance improvements in stdlib that are problematic for pypy or others? Personally I would like cleaner code in stdlib vs speeding up CPython. Typically that also helps pypy so I'm not unbiased. Cheers, fijal

21 61

what is a dict_keys and where can I import it from?
by Chris Withers Feb. 12, 2013

Feb. 12, 2013

Hi all, So, dicts in Python 3 return "something different" from their keys and values methods: >>> dict(x=1, y=2).keys() dict_keys(['y', 'x']) >>> type(dict(x=1, y=2).keys()) <class 'dict_keys'> I have vague memories of these things being referred to as views or some such? Where can I learn more? More importantly, how can I tell if I have one of them? I guess I can keep a reference to type({}.keys()) somewhere, but that feels a bit yucky. Should these things be in the types module? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

4 3

Submitting PEP 422 (Simple class initialization hook) for pronouncement
by Nick Coghlan Feb. 12, 2013

Feb. 12, 2013

I wrote PEP 422 (simple class initialization hook) last year after PJE pointed out that Python 3 style metaclasses made some Python 2 code impossible to migrate (since the class body could no longer request modifications to be made to the class after initialization was complete). It then languished, as I never found the time to actually implement it. Fortunately, Daniel Urban has now agreed to be a co-author on the PEP, and has fixed a couple of lingering technical errors in the PEP, as well as providing a reference implementation at http://bugs.python.org/issue17044 For those that don't recall the original discussion, the proposal is to add a new __init_class__ hook, invoked after the class object is created, but before the class decorators are applied. This provides a simple approach to inherited post-creation modification of classes, without the need for a custom metaclass. There's currently no BDFL delegate assigned, so this is a request for a pronouncement from Guido. Regards, Nick. ======================================= PEP: 422 Title: Simple class initialisation hook Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Daniel Urban <urban.dani+py(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-Jun-2012 Python-Version: 3.4 Post-History: 5-Jun-2012, 10-Feb-2012 Abstract ======== In Python 2, the body of a class definition could modify the way a class was created (or simply arrange to run other code after the class was created) by setting the ``__metaclass__`` attribute in the class body. While doing this implicitly from called code required the use of an implementation detail (specifically, ``sys._getframes()``), it could also be done explicitly in a fully supported fashion (for example, by passing ``locals()`` to a function that calculated a suitable ``__metaclass__`` value) There is currently no corresponding mechanism in Python 3 that allows the code executed in the class body to directly influence how the class object is created. Instead, the class creation process is fully defined by the class header, before the class body even begins executing. This PEP proposes a mechanism that will once again allow the body of a class definition to more directly influence the way a class is created (albeit in a more constrained fashion), as well as replacing some current uses of metaclasses with a simpler, easier to understand alternative. Background ========== For an already created class ``cls``, the term "metaclass" has a clear meaning: it is the value of ``type(cls)``. *During* class creation, it has another meaning: it is also used to refer to the metaclass hint that may be provided as part of the class definition. While in many cases these two meanings end up referring to one and the same object, there are two situations where that is not the case: * If the metaclass hint refers to an instance of ``type``, then it is considered as a candidate metaclass along with the metaclasses of all of the parents of the class being defined. If a more appropriate metaclass is found amongst the candidates, then it will be used instead of the one given in the metaclass hint. * Otherwise, an explicit metaclass hint is assumed to be a factory function and is called directly to create the class object. In this case, the final metaclass will be determined by the factory function definition. In the typical case (where the factory functions just calls ``type``, or, in Python 3.3 or later, ``types.new_class``) the actual metaclass is then determined based on the parent classes. It is notable that only the actual metaclass is inherited - a factory function used as a metaclass hook sees only the class currently being defined, and is not invoked for any subclasses. In Python 3, the metaclass hint is provided using the ``metaclass=Meta`` keyword syntax in the class header. This allows the ``__prepare__`` method on the metaclass to be used to create the ``locals()`` namespace used during execution of the class body (for example, specifying the use of ``collections.OrderedDict`` instead of a regular ``dict``). In Python 2, there was no ``__prepare__`` method (that API was added for Python 3 by PEP 3115). Instead, a class body could set the ``__metaclass__`` attribute, and the class creation process would extract that value from the class namespace to use as the metaclass hint. There is `published code`_ that makes use of this feature. Another new feature in Python 3 is the zero-argument form of the ``super()`` builtin, introduced by PEP 3135. This feature uses an implicit ``__class__`` reference to the class being defined to replace the "by name" references required in Python 2. Just as code invoked during execution of a Python 2 metaclass could not call methods that referenced the class by name (as the name had not yet been bound in the containing scope), similarly, Python 3 metaclasses cannot call methods that rely on the implicit ``__class__`` reference (as it is not populated until after the metaclass has returned control to the class creation machiner). Proposal ======== This PEP proposes that a mechanism be added to Python 3 that meets the following criteria: 1. Restores the ability for class namespaces to have some influence on the class creation process (above and beyond populating the namespace itself), but potentially without the full flexibility of the Python 2 style ``__metaclass__`` hook 2. Integrates nicely with class inheritance structures (including mixins and multiple inheritance) 3. Integrates nicely with the implicit ``__class__`` reference and zero-argument ``super()`` syntax introduced by PEP 3135 4. Can be added to an existing base class without a significant risk of introducing backwards compatibility problems One mechanism that can achieve this goal is to add a new class initialisation hook, modelled directly on the existing instance initialisation hook, but with the signature constrained to match that of an ordinary class decorator. Specifically, it is proposed that class definitions be able to provide a class initialisation hook as follows:: class Example: @classmethod def __init_class__(cls): # This is invoked after the class is created, but before any # explicit decorators are called # The usual super() mechanisms are used to correctly support # multiple inheritance. The class decorator style signature helps # ensure that invoking the parent class is as simple as possible. If present on the created object, this new hook will be called by the class creation machinery *after* the ``__class__`` reference has been initialised. For ``types.new_class()``, it will be called as the last step before returning the created class object. If a metaclass wishes to block class initialisation for some reason, it must arrange for ``cls.__init_class__`` to trigger ``AttributeError``. Note, that when ``__init_class__`` is called, the name of the class is not bound to the new class object yet. As a consequence, the two argument form of ``super()`` cannot be used to call methods (e.g., ``super(Example, cls)`` wouldn't work in the example above). However, the zero argument form of ``super()`` works as expected, since the ``__class__`` reference is already initialised. This general proposal is not a new idea (it was first suggested for inclusion in the language definition `more than 10 years ago`_, and a similar mechanism has long been supported by `Zope's ExtensionClass`_), but the situation has changed sufficiently in recent years that the idea is worth reconsidering. Key Benefits ============ Replaces many use cases for dynamic setting of ``__metaclass__`` ----------------------------------------------------------------- For use cases that don't involve completely replacing the defined class, Python 2 code that dynamically set ``__metaclass__`` can now dynamically set ``__init_class__`` instead. For more advanced use cases, introduction of an explicit metaclass (possibly made available as a required base class) will still be necessary in order to support Python 3. Easier inheritance of definition time behaviour ----------------------------------------------- Understanding Python's metaclasses requires a deep understanding of the type system and the class construction process. This is legitimately seen as challenging, due to the need to keep multiple moving parts (the code, the metaclass hint, the actual metaclass, the class object, instances of the class object) clearly distinct in your mind. Even when you know the rules, it's still easy to make a mistake if you're not being extremely careful. An earlier version of this PEP actually included such a mistake: it stated "subclass of type" for a constraint that is actually "instance of type". Understanding the proposed class initialisation hook only requires understanding decorators and ordinary method inheritance, which isn't quite as daunting a task. The new hook provides a more gradual path towards understanding all of the phases involved in the class definition process. Reduced chance of metaclass conflicts ------------------------------------- One of the big issues that makes library authors reluctant to use metaclasses (even when they would be appropriate) is the risk of metaclass conflicts. These occur whenever two unrelated metaclasses are used by the desired parents of a class definition. This risk also makes it very difficult to *add* a metaclass to a class that has previously been published without one. By contrast, adding an ``__init_class__`` method to an existing type poses a similar level of risk to adding an ``__init__`` method: technically, there is a risk of breaking poorly implemented subclasses, but when that occurs, it is recognised as a bug in the subclass rather than the library author breaching backwards compatibility guarantees. In fact, due to the constrained signature of ``__init_class__``, the risk in this case is actually even lower than in the case of ``__init__``. Integrates cleanly with \PEP 3135 --------------------------------- Unlike code that runs as part of the metaclass, code that runs as part of the new hook will be able to freely invoke class methods that rely on the implicit ``__class__`` reference introduced by PEP 3135, including methods that use the zero argument form of ``super()``. Alternatives ============ The Python 3 Status Quo ----------------------- The Python 3 status quo already offers a great deal of flexibility. For changes which only affect a single class definition and which can be specified at the time the code is written, then class decorators can be used to modify a class explicitly. Class decorators largely ignore class inheritance and can make full use of methods that rely on the ``__class__`` reference being populated. Using a custom metaclass provides the same level of power as it did in Python 2. However, it's notable that, unlike class decorators, a metaclass cannot call any methods that rely on the ``__class__`` reference, as that reference is not populated until after the metaclass constructor returns control to the class creation code. One major use case for metaclasses actually closely resembles the use of class decorators. It occurs whenever a metaclass has an implementation that uses the following pattern:: class Metaclass(type): def __new__(meta, *args, **kwds): cls = super(Metaclass, meta).__new__(meta, *args, **kwds) # Do something with cls return cls The key difference between this pattern and a class decorator is that it is automatically inherited by subclasses. However, it also comes with a major disadvantage: Python does not allow you to inherit from classes with unrelated metaclasses. Thus, the status quo requires that developers choose between the following two alternatives: * Use a class decorator, meaning that behaviour is not inherited and must be requested explicitly on every subclass * Use a metaclass, meaning that behaviour is inherited, but metaclass conflicts may make integration with other libraries and frameworks more difficult than it otherwise would be If this PEP is ultimately rejected, then this is the existing design that will remain in place by default. Restoring the Python 2 metaclass hook ------------------------------------- One simple alternative would be to restore support for a Python 2 style ``metaclass`` hook in the class body. This would be checked after the class body was executed, potentially overwriting the metaclass hint provided in the class header. The main attraction of such an approach is that it would simplify porting Python 2 applications that make use of this hook (especially those that do so dynamically). However, this approach does nothing to simplify the process of adding *inherited* class definition time behaviour, nor does it interoperate cleanly with the PEP 3135 ``__class__`` and ``super()`` semantics (as with any metaclass based solution, the ``__metaclass__`` hook would have to run before the ``__class__`` reference has been populated. Dynamic class decorators ------------------------ The original version of this PEP was called "Dynamic class decorators" and focused solely on a significantly more complicated proposal than that presented in the current version. As with the current version, it proposed that a new step be added to the class creation process, after the metaclass invocation to construct the class instance and before the application of lexical decorators. However, instead of a simple process of calling a single class method that relies on normal inheritance mechanisms, it proposed a far more complicated procedure that walked the class MRO looking for decorators stored in iterable ``__decorators__`` attributes. Using the current version of the PEP, the scheme originally proposed could be implemented as:: class DynamicDecorators(Base): @classmethod def __init_class__(cls): # Process any classes later in the MRO try: mro_chain = super().__init_class__ except AttributeError: pass else: mro_chain() # Process any __decorators__ attributes in the MRO for entry in reversed(cls.mro()): decorators = entry.__dict__.get("__decorators__", ()) for deco in reversed(decorators): cls = deco(cls) Any subclasses of ``DynamicDecorators`` would then automatically have the contents of any ``__decorators__`` attributes processed and invoked. The mechanism in the current PEP is considered superior, as many issues to do with ordering and the same decorator being invoked multiple times just go away, as that kind of thing is taken care of through the use of an ordinary class method invocation. Automatic metaclass derivation ------------------------------ When no appropriate metaclass is found, it's theoretically possible to automatically derive a metaclass for a new type based on the metaclass hint and the metaclasses of the bases. While adding such a mechanism would reduce the risk of spurious metaclass conflicts, it would do nothing to improve integration with PEP 3135, would not help with porting Python 2 code that set ``__metaclass__`` dynamically and would not provide a more straightforward inherited mechanism for invoking additional operations after the class invocation is complete. In addition, there would still be a risk of metaclass conflicts in cases where the base metaclasses were not written with multiple inheritance in mind. In such situations, there's a chance of introducing latent defects if one or more metaclasses are not invoked correctly. Calling the new hook from ``type.__init__`` ------------------------------------------- Calling the new hook automatically from ``type.__init__``, would achieve most of the goals of this PEP. However, using that approach would mean that ``__init_class__`` implementations would be unable to call any methods that relied on the ``__class__`` reference (or used the zero-argument form of ``super()``), and could not make use of those features themselves. Reference Implementation ======================== The reference implementation has been posted to the `issue tracker`_. References ========== .. _published code: http://mail.python.org/pipermail/python-dev/2012-June/119878.html .. _more than 10 years ago: http://mail.python.org/pipermail/python-dev/2001-November/018651.html .. _Zope's ExtensionClass: http://docs.zope.org/zope_secrets/extensionclass.html .. _issue tracker: http://bugs.python.org/issue17044 Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

10 25

My CLA
by anatoly techtonik Feb. 11, 2013

Feb. 11, 2013

Python Contributor Agreement ---------------------------- I allow PSF to release all my code that I submitted to it, under any open source license. -- anatoly t.

7 9

checking what atexit handlers are registered in Python 3
by Chris Withers Feb. 11, 2013

Feb. 11, 2013

Hi All, I'm finally getting around to porting some of the packages I maintain over to Python 3. One rough edge I've hit: I see the atexit module has moved to be C-based and, as far as I can tell, no longer allows you to introspect what atexit functions have been registered. If I'm writing tests for code that registers atexit handlers, how can I check that they've been correctly registered? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

2 1

Re: [Python-Dev] [Python-checkins] cpython (3.3): Reject float as uid or gid.
by Eric V. Smith Feb. 11, 2013

Feb. 11, 2013

On 2/10/2013 4:29 PM, serhiy.storchaka wrote: > http://hg.python.org/cpython/rev/4ef048f4834e > changeset: 82147:4ef048f4834e > branch: 3.3 > parent: 82145:b322655a4a88 > user: Serhiy Storchaka <storchaka(a)gmail.com> > date: Sun Feb 10 23:28:02 2013 +0200 > summary: > Reject float as uid or gid. > A regression was introduced in the commit for issue issue #4591. > > files: > Modules/posixmodule.c | 16 ++++++++++++++-- > 1 files changed, 14 insertions(+), 2 deletions(-) > > > diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c > --- a/Modules/posixmodule.c > +++ b/Modules/posixmodule.c > @@ -437,7 +437,13 @@ > _Py_Uid_Converter(PyObject *obj, void *p) > { > int overflow; > - long result = PyLong_AsLongAndOverflow(obj, &overflow); > + long result; > + if (PyFloat_Check(obj)) { > + PyErr_SetString(PyExc_TypeError, > + "integer argument expected, got float"); > + return 0; > + } > + result = PyLong_AsLongAndOverflow(obj, &overflow); > if (overflow < 0) > goto OverflowDown; > if (!overflow && result == -1) { Instead of special-casing float, isn't using __index__ the preferred way to do this? -- Eric.

2 1