Mailman 3 June 2016 - Python-Dev

Summary of Python tracker Issues
by Python tracker 24 Jun '16

24 Jun '16

ACTIVITY SUMMARY (2016-06-17 - 2016-06-24) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5531 (-13) closed 33609 (+52) total 39140 (+39) Open issues with patches: 2419 Issues opened (27) ================== #21106: Updated Mac folder icon http://bugs.python.org/issue21106 reopened by r.david.murray #27196: Eliminate 'ThemeChanged' warning when running IDLE tests http://bugs.python.org/issue27196 reopened by ned.deily #27346: Implement os.readv() / os.writev() http://bugs.python.org/issue27346 opened by mmarkk #27348: Non-main thread exception handler drops exception message http://bugs.python.org/issue27348 opened by martin.panter #27350: Compact and ordered dict http://bugs.python.org/issue27350 opened by naoki #27351: Unexpected ConfigParser.read() behavior when passed fileobject http://bugs.python.org/issue27351 opened by Rich.Rauenzahn #27352: Bug in IMPORT_NAME http://bugs.python.org/issue27352 opened by serhiy.storchaka #27353: Add nroot function to math http://bugs.python.org/issue27353 opened by steven.daprano #27354: SSLContext.load_verify_locations cannot handle paths on Window http://bugs.python.org/issue27354 opened by Ilya.Kulakov #27355: Strip out the last lingering vestiges of Windows CE support http://bugs.python.org/issue27355 opened by larry #27357: Enhancing the Windows installer http://bugs.python.org/issue27357 opened by Sworddragon #27358: BUILD_MAP_UNPACK_WITH_CALL is slow http://bugs.python.org/issue27358 opened by Demur Rumed #27359: OrderedDict pseudo-literals (WIP) http://bugs.python.org/issue27359 opened by llllllllll #27363: Complex with negative zero imaginary part http://bugs.python.org/issue27363 opened by Vedran.Čačić #27364: Deprecate invalid unicode escape sequences http://bugs.python.org/issue27364 opened by ebarry #27365: Allow non-ascii chars in IDLE NEWS.txt (for contributor names) http://bugs.python.org/issue27365 opened by terry.reedy #27366: PEP487: Simpler customization of class creation http://bugs.python.org/issue27366 opened by Martin.Teichmann #27367: Windows buildbot: random timeout failure on test_threading http://bugs.python.org/issue27367 opened by haypo #27369: [PATCH] Tests break with --with-system-expat and Expat 2.2.0 http://bugs.python.org/issue27369 opened by sping #27372: Test_idle should stop changing locale http://bugs.python.org/issue27372 opened by terry.reedy #27373: logging.handlers.SysLogHandler with TCP not working on rsyslog http://bugs.python.org/issue27373 opened by 吴冉波 #27374: Cygwin: Makefile does not install DLL import library http://bugs.python.org/issue27374 opened by erik.bray #27376: Add mock_import method to mock module http://bugs.python.org/issue27376 opened by Eyal Posener #27377: Add smarter socket.fromfd() http://bugs.python.org/issue27377 opened by nascheme #27379: SocketType changed in Python 3 http://bugs.python.org/issue27379 opened by martin.panter #27380: IDLE: add base Query dialog with ttk widgets http://bugs.python.org/issue27380 opened by terry.reedy #27383: executuable in distutils triggering microsoft anti virus http://bugs.python.org/issue27383 opened by Rob Bairos Most recent 15 issues with no replies (15) ========================================== #27383: executuable in distutils triggering microsoft anti virus http://bugs.python.org/issue27383 #27380: IDLE: add base Query dialog with ttk widgets http://bugs.python.org/issue27380 #27379: SocketType changed in Python 3 http://bugs.python.org/issue27379 #27376: Add mock_import method to mock module http://bugs.python.org/issue27376 #27374: Cygwin: Makefile does not install DLL import library http://bugs.python.org/issue27374 #27372: Test_idle should stop changing locale http://bugs.python.org/issue27372 #27367: Windows buildbot: random timeout failure on test_threading http://bugs.python.org/issue27367 #27366: PEP487: Simpler customization of class creation http://bugs.python.org/issue27366 #27348: Non-main thread exception handler drops exception message http://bugs.python.org/issue27348 #27340: bytes-like objects with socket.sendall(), SSL, and http.client http://bugs.python.org/issue27340 #27332: Clinic: first parameter for module-level functions should be P http://bugs.python.org/issue27332 #27331: Add a policy argument to email.mime.MIMEBase http://bugs.python.org/issue27331 #27329: Document behavior when CDLL is called with None as an argumen http://bugs.python.org/issue27329 #27326: SIGSEV in test_window_funcs of test_curses http://bugs.python.org/issue27326 #27323: ncurses putwin() fails in test_module_funcs http://bugs.python.org/issue27323 Most recent 15 issues waiting for review (15) ============================================= #27380: IDLE: add base Query dialog with ttk widgets http://bugs.python.org/issue27380 #27377: Add smarter socket.fromfd() http://bugs.python.org/issue27377 #27376: Add mock_import method to mock module http://bugs.python.org/issue27376 #27374: Cygwin: Makefile does not install DLL import library http://bugs.python.org/issue27374 #27372: Test_idle should stop changing locale http://bugs.python.org/issue27372 #27369: [PATCH] Tests break with --with-system-expat and Expat 2.2.0 http://bugs.python.org/issue27369 #27366: PEP487: Simpler customization of class creation http://bugs.python.org/issue27366 #27365: Allow non-ascii chars in IDLE NEWS.txt (for contributor names) http://bugs.python.org/issue27365 #27364: Deprecate invalid unicode escape sequences http://bugs.python.org/issue27364 #27359: OrderedDict pseudo-literals (WIP) http://bugs.python.org/issue27359 #27358: BUILD_MAP_UNPACK_WITH_CALL is slow http://bugs.python.org/issue27358 #27355: Strip out the last lingering vestiges of Windows CE support http://bugs.python.org/issue27355 #27352: Bug in IMPORT_NAME http://bugs.python.org/issue27352 #27350: Compact and ordered dict http://bugs.python.org/issue27350 #27334: pysqlite3 context manager not performing rollback when a datab http://bugs.python.org/issue27334 Top 10 most discussed issues (10) ================================= #27365: Allow non-ascii chars in IDLE NEWS.txt (for contributor names) http://bugs.python.org/issue27365 19 msgs #27051: Create PIP gui http://bugs.python.org/issue27051 15 msgs #27350: Compact and ordered dict http://bugs.python.org/issue27350 9 msgs #27172: Undeprecate inspect.getfullargspec() http://bugs.python.org/issue27172 8 msgs #27344: zipfile *does* support utf-8 filenames http://bugs.python.org/issue27344 8 msgs #27309: Visual Styles support to tk/tkinter file and message dialogs http://bugs.python.org/issue27309 7 msgs #27353: Add nroot function to math http://bugs.python.org/issue27353 7 msgs #27359: OrderedDict pseudo-literals (WIP) http://bugs.python.org/issue27359 7 msgs #27364: Deprecate invalid unicode escape sequences http://bugs.python.org/issue27364 7 msgs #27363: Complex with negative zero imaginary part http://bugs.python.org/issue27363 6 msgs Issues closed (50) ================== #5225: OS X "Update Shell Profile" may not update $PATH if run more t http://bugs.python.org/issue5225 closed by willingc #8406: Make some setup.py paths exclude-able http://bugs.python.org/issue8406 closed by willingc #9156: socket._fileobject: read raises AttributeError when closed in http://bugs.python.org/issue9156 closed by martin.panter #11623: Distutils is reporting OSX 10.6 w/ XCode 4 as "universal" http://bugs.python.org/issue11623 closed by willingc #14354: Crash in _ctypes_alloc_callback http://bugs.python.org/issue14354 closed by willingc #16821: bundlebuilder broken in 2.7 http://bugs.python.org/issue16821 closed by willingc #18300: script_helper._assert_python should set TERM='' by default. http://bugs.python.org/issue18300 closed by berker.peksag #20256: Argument Clinic: compare signed and unsigned ints http://bugs.python.org/issue20256 closed by serhiy.storchaka #22463: Warnings when building on AIX http://bugs.python.org/issue22463 closed by martin.panter #22636: avoid using a shell in ctypes.util: replace os.popen with subp http://bugs.python.org/issue22636 closed by martin.panter #23641: Got rid of bad dunder names http://bugs.python.org/issue23641 closed by serhiy.storchaka #24137: Force not using _default_root in IDLE http://bugs.python.org/issue24137 closed by terry.reedy #24314: irrelevant cross-link in documentation of user-defined functio http://bugs.python.org/issue24314 closed by martin.panter #24419: In argparse action append_const doesn't work for positional ar http://bugs.python.org/issue24419 closed by paul.j3 #26290: fileinput and 'for line in sys.stdin' do strange mockery of in http://bugs.python.org/issue26290 closed by martin.panter #26536: Add the SIO_LOOPBACK_FAST_PATH option to socket.ioctl http://bugs.python.org/issue26536 closed by steve.dower #26547: Undocumented use of the term dictproxy in vars() documentation http://bugs.python.org/issue26547 closed by berker.peksag #26930: Upgrade installers to OpenSSL 1.0.2h http://bugs.python.org/issue26930 closed by steve.dower #26975: Decimal.from_float works incorrectly for non-binary floats http://bugs.python.org/issue26975 closed by skrah #27006: C implementation of Decimal.from_float() bypasses __new__ and http://bugs.python.org/issue27006 closed by skrah #27021: It is not documented that os.writev() suffer from SC_IOV_MAX http://bugs.python.org/issue27021 closed by orsenthil #27048: distutils._msvccompiler._get_vc_env() fails with UnicodeDecode http://bugs.python.org/issue27048 closed by steve.dower #27079: Bugs in curses.ascii predicates http://bugs.python.org/issue27079 closed by serhiy.storchaka #27177: re match.group should support __index__ http://bugs.python.org/issue27177 closed by serhiy.storchaka #27244: print(';;') fails in pdb with SyntaxError http://bugs.python.org/issue27244 closed by ned.deily #27287: SIGSEGV when calling os.forkpty() http://bugs.python.org/issue27287 closed by ned.deily #27294: Better repr for Tkinter event objects http://bugs.python.org/issue27294 closed by serhiy.storchaka #27297: Add support for /dev/random to "secrets" http://bugs.python.org/issue27297 closed by ncoghlan #27299: urllib does not splitport while putrequest realhost to HTTP he http://bugs.python.org/issue27299 closed by r.david.murray #27304: Create "Source Code" links in module sections, where relevant http://bugs.python.org/issue27304 closed by terry.reedy #27312: test_setupapp (idlelib.idle_test.test_macosx.SetupTest) fails http://bugs.python.org/issue27312 closed by ned.deily #27319: Multiple item arguments for selection operations http://bugs.python.org/issue27319 closed by serhiy.storchaka #27333: validate_step in rangeobject.c, incorrect code logic but right http://bugs.python.org/issue27333 closed by serhiy.storchaka #27337: 3.6.0a2 tarball has weird paths http://bugs.python.org/issue27337 closed by ned.deily #27342: Clean up some Py_XDECREFs in rangeobject.c and bltinmodule.c http://bugs.python.org/issue27342 closed by serhiy.storchaka #27343: Incorrect error message for conflicting initializers of ctypes http://bugs.python.org/issue27343 closed by serhiy.storchaka #27345: GzipFile's readinto() reads gzip data instead of file data. http://bugs.python.org/issue27345 closed by Ryan Birmingham #27347: Spam http://bugs.python.org/issue27347 closed by berker.peksag #27349: distutils.command.upload: typo "protcol_version" http://bugs.python.org/issue27349 closed by berker.peksag #27356: csv http://bugs.python.org/issue27356 closed by berker.peksag #27360: _deque and _islice are sometimes None http://bugs.python.org/issue27360 closed by SilentGhost #27361: ValueError on eval after 'from pandas import *' http://bugs.python.org/issue27361 closed by SilentGhost #27362: json.dumps to check for obj.__json__ before raising TypeError http://bugs.python.org/issue27362 closed by r.david.murray #27368: os.mkdir is not working for multiple level of directory creati http://bugs.python.org/issue27368 closed by SilentGhost #27370: Inconsistency in docs for list.extend http://bugs.python.org/issue27370 closed by martin.panter #27371: Runaway memory consumption using tkinter update() http://bugs.python.org/issue27371 closed by jeremyblow #27375: error in "make test", while trying to install python on linux http://bugs.python.org/issue27375 closed by skrah #27378: remove ref to Phil Schwartz's 'Kodos' in regex HOWTO http://bugs.python.org/issue27378 closed by berker.peksag #27381: Typo in zipfile documentation http://bugs.python.org/issue27381 closed by berker.peksag #27382: calendar module .isleap() probleam http://bugs.python.org/issue27382 closed by ebarry

1 0

Here's what's going into 3.5.2 final and 3.4.5 final
by Larry Hastings 24 Jun '16

24 Jun '16

Heads up! This is a courtesy reminder from your friendly 3.4 and 3.5 release manager. Here's a list of all the changes since 3.5.2rc1 that are currently going into 3.5.2 final: * 155e665428c6 - Zachary: OpenSSL 1.0.2h build changes for Windows * cae0b7ffeb9f - Benjamin: fix % in Doc/whatsnew/3.5.rst that confuses latex * 783dfd77e4c1 - Terry Reed: allow non-ascii in idlelib/NEWS.txt * <new revision> - Matthias: fix for test_ssl test_options on Ubuntu 3.4.5 final only has one change from 3.4.5rc1: the test_ssl test_options fix from Matthias. If there's something else that needs to go into one of these releases, and it's not on the list above, speak up now. I may actually tag these late Friday as I'm traveling Saturday. So you have approximately, let's say 20 hours from when I post this. Thanks, //arry/

1 0

Why are class dictionaries not accessible?
by Random832 23 Jun '16

23 Jun '16

The documentation states: """Objects such as modules and instances have an updateable __dict__ attribute; however, other objects may have write restrictions on their __dict__ attributes (for example, classes use a dictproxy to prevent direct dictionary updates).""" However, it's not clear from that *why* direct dictionary updates are undesirable. This not only prevents you from getting a reference to the real class dict (which is the apparent goal), but is also the fundamental reason why you can't use a metaclass to put, say, an OrderedDict in its place - because the type constructor has to copy the dict that was used in class preparation into a new dict rather than using the one that was actually returned by __prepare__. [Also, the name of the type used for this is mappingproxy, not dictproxy]

3 5

PEP 487: Simpler customization of class creation
by Martin Teichmann 22 Jun '16

22 Jun '16

Hi list, using metaclasses in Python is a very flexible method of customizing class creation, yet this customization comes at a cost: once you want to combine two classes with different metaclasses, you run into problems. This is why I proposed PEP 487 (see https://github.com/tecki/peps/blob/pep487/pep-0487.txt, which I also attached here for ease of discussion), proposing a simple hook into class creation, with which one can override in subclasses such that sub-subclasses get customized accordingly. Otherwise, the standard Python inheritance rules apply (super() and the MRO). I also proposed to store the order in which attributes in classes are defined. This is exactly the same as PEP 520, discussed here recently, just that unfortunately we chose different names, but I am open for suggestions for better names. After having gotten good feedback on python-ideas (see https://mail.python.org/pipermail/python-ideas/2016-February/038305.html) and from IPython traitlets as a potential user of the feature (see https://mail.scipy.org/pipermail/ipython-dev/2016-February/017066.html, and the code at https://github.com/tecki/traitlets/commits/pep487) I implemented a pure Python version of this PEP, to be introduced into the standard library. I also wrote a proof-of-concept for another potential user of this feature, django forms, at https://github.com/tecki/django/commits/no-metaclass. The code to be introduced into the standard library can be found at https://github.com/tecki/cpython/commits/pep487 (sorry for using github, I'll submit something using hg once I understand that toolchain). It introduces a new metaclass types.Type which contains the machinery, and a new base class types.Object which uses said metaclass. The naming was chosen to clarify the intention that eventually those classes may be implemented in C and replace type and object. As above, I am open to better naming. As a second step, I let abc.ABCMeta inherit from said types.Type, such that an ABC may also use the features of my metaclass, without the need to define a new mixing metaclass. I am looking forward to a lot of comments on this! Greetings Martin The proposed PEP for discussion: PEP: 487 Title: Simpler customisation of class creation Version: $Revision$ Last-Modified: $Date$ Author: Martin Teichmann <lkb.teichmann(a)gmail.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Feb-2015 Python-Version: 3.6 Post-History: 27-Feb-2015, 5-Feb-2016 Replaces: 422 Abstract ======== Currently, customising class creation requires the use of a custom metaclass. This custom metaclass then persists for the entire lifecycle of the class, creating the potential for spurious metaclass conflicts. This PEP proposes to instead support a wide range of customisation scenarios through a new ``__init_subclass__`` hook in the class body, a hook to initialize attributes, and a way to keep the order in which attributes are defined. Those hooks should at first be defined in a metaclass in the standard library, with the option that this metaclass eventually becomes the default ``type`` metaclass. The new mechanism should be easier to understand and use than implementing a custom metaclass, and thus should provide a gentler introduction to the full power Python's metaclass machinery. Background ========== Metaclasses are a powerful tool to customize class creation. They have, however, the problem that there is no automatic way to combine metaclasses. If one wants to use two metaclasses for a class, a new metaclass combining those two needs to be created, typically manually. This need often occurs as a surprise to a user: inheriting from two base classes coming from two different libraries suddenly raises the necessity to manually create a combined metaclass, where typically one is not interested in those details about the libraries at all. This becomes even worse if one library starts to make use of a metaclass which it has not done before. While the library itself continues to work perfectly, suddenly every code combining those classes with classes from another library fails. Proposal ======== While there are many possible ways to use a metaclass, the vast majority of use cases falls into just three categories: some initialization code running after class creation, the initalization of descriptors and keeping the order in which class attributes were defined. Those three use cases can easily be performed by just one metaclass. If this metaclass is put into the standard library, and all libraries that wish to customize class creation use this very metaclass, no combination of metaclasses is necessary anymore. Said metaclass should live in the ``types`` module under the name ``Type``. This should hint the user that in the future, this metaclass may become the default metaclass ``type``. The three use cases are achieved as follows: 1. The metaclass contains an ``__init_subclass__`` hook that initializes all subclasses of a given class, 2. the metaclass calls a ``__set_owner__`` hook on all the attribute (descriptors) defined in the class, and 3. an ``__attribute_order__`` tuple is left in the class in order to inspect the order in which attributes were defined. For ease of use, a base class ``types.Object`` is defined, which uses said metaclass and contains an empty stub for the hook described for use case 1. It will eventually become the new replacement for the standard ``object``. As an example, the first use case looks as follows:: >>> class SpamBase(types.Object): ... # this is implicitly a @classmethod ... def __init_subclass__(cls, **kwargs): ... cls.class_args = kwargs ... super().__init_subclass__(cls, **kwargs) >>> class Spam(SpamBase, a=1, b="b"): ... pass >>> Spam.class_args {'a': 1, 'b': 'b'} The base class ``types.Object`` contains an empty ``__init_subclass__`` method which serves as an endpoint for cooperative multiple inheritance. Note that this method has no keyword arguments, meaning that all methods which are more specialized have to process all keyword arguments. This general proposal is not a new idea (it was first suggested for inclusion in the language definition `more than 10 years ago`_, and a similar mechanism has long been supported by `Zope's ExtensionClass`_), but the situation has changed sufficiently in recent years that the idea is worth reconsidering for inclusion. The second part of the proposal adds an ``__set_owner__`` initializer for class attributes, especially if they are descriptors. Descriptors are defined in the body of a class, but they do not know anything about that class, they do not even know the name they are accessed with. They do get to know their owner once ``__get__`` is called, but still they do not know their name. This is unfortunate, for example they cannot put their associated value into their object's ``__dict__`` under their name, since they do not know that name. This problem has been solved many times, and is one of the most important reasons to have a metaclass in a library. While it would be easy to implement such a mechanism using the first part of the proposal, it makes sense to have one solution for this problem for everyone. To give an example of its usage, imagine a descriptor representing weak referenced values:: import weakref class WeakAttribute: def __get__(self, instance, owner): return instance.__dict__[self.name] def __set__(self, instance, value): instance.__dict__[self.name] = weakref.ref(value) # this is the new initializer: def __set_owner__(self, owner, name): self.name = name While this example looks very trivial, it should be noted that until now such an attribute cannot be defined without the use of a metaclass. And given that such a metaclass can make life very hard, this kind of attribute does not exist yet. The third part of the proposal is to leave a tuple called ``__attribute_order__`` in the class that contains the order in which the attributes were defined. This is a very common usecase, many libraries use an ``OrderedDict`` to store this order. This is a very simple way to achieve the same goal. Under the hood, the implementation *does* ``__prepare__`` an ``OrderedDict`` namespace, it just retains the order of the keys in ``__attribute_order__``, since ``type.__new__`` will cripple the ``OrderedDict`` into a normal ``dict``, discarding the order information. Key Benefits ============ Easier inheritance of definition time behaviour ----------------------------------------------- Understanding Python's metaclasses requires a deep understanding of the type system and the class construction process. This is legitimately seen as challenging, due to the need to keep multiple moving parts (the code, the metaclass hint, the actual metaclass, the class object, instances of the class object) clearly distinct in your mind. Even when you know the rules, it's still easy to make a mistake if you're not being extremely careful. Understanding the proposed implicit class initialization hook only requires ordinary method inheritance, which isn't quite as daunting a task. The new hook provides a more gradual path towards understanding all of the phases involved in the class definition process. Reduced chance of metaclass conflicts ------------------------------------- One of the big issues that makes library authors reluctant to use metaclasses (even when they would be appropriate) is the risk of metaclass conflicts. These occur whenever two unrelated metaclasses are used by the desired parents of a class definition. This risk also makes it very difficult to *add* a metaclass to a class that has previously been published without one. By contrast, adding an ``__init_subclass__`` method to an existing type poses a similar level of risk to adding an ``__init__`` method: technically, there is a risk of breaking poorly implemented subclasses, but when that occurs, it is recognised as a bug in the subclass rather than the library author breaching backwards compatibility guarantees. A path of introduction into Python ================================== Most of the benefits of this PEP can already be implemented using a simple metaclass. For the ``__init_subclass__`` hook this works all the way down to Python 2.7, while the attribute order needs Python 3.0 to work. Such a class has been `uploaded to PyPI`_. The only drawback of such a metaclass are the mentioned problems with metaclasses and multiple inheritance. Two classes using such a metaclass can only be combined, if they use exactly the same such metaclass. This fact calls for the inclusion of such a class into the standard library, as ``types.Type``, with a ``types.Object`` base class using it. Once all users use this standard library metaclass, classes from different packages can easily be combined. But still such classes cannot be easily combined with other classes using other metaclasses. Authors of metaclasses should bear that in mind and inherit from the standard metaclass if it seems useful for users of the metaclass to add more functionality. Ultimately, if the need for combining with other metaclasses is strong enough, the proposed functionality may be introduced into Python's ``type``. Those arguments strongly hint to the following procedure to include the proposed functionality into Python: 1. The metaclass implementing this proposal is put onto PyPI, so that it can be used and scrutinized. 2. Introduce this class into the Python 3.6 standard library. 3. Consider this as the default behavior for Python 3.7. Steps 2 and 3 would be similar to how the ``set`` datatype was first introduced as ``sets.Set``, and only later made a builtin type (with a slightly different API) based on wider experiences with the ``sets`` module. While the metaclass is still in the standard library and not in the language, it may still clash with other metaclasses. The most prominent metaclass in use is probably ABCMeta. It is also a particularly good example for the need of combining metaclasses. For users who want to define a ABC with subclass initialization, we should support a ``types.ABCMeta`` class, or let ``abc.ABCMeta`` inherit from this PEP's metaclass. As it turns out, most of the behavior of ``abc.ABCMeta`` can be done achieved with our ``types.Type``, except its core behavior, ``__instancecheck__`` and ``__subclasscheck__`` which can be supplied, as per the definition of the Python language, exclusively in a metaclass. Extensions written in C or C++ also often define their own metaclass. It would be very useful if those could also inherit from the metaclass defined here, but this is probably not possible. New Ways of Using Classes ========================= This proposal has many usecases like the following. In the examples, we still inherit from the ``SubclassInit`` base class. This would become unnecessary once this PEP is included in Python directly. Subclass registration --------------------- Especially when writing a plugin system, one likes to register new subclasses of a plugin baseclass. This can be done as follows:: class PluginBase(Object): subclasses = [] def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) cls.subclasses.append(cls) In this example, ``PluginBase.subclasses`` will contain a plain list of all subclasses in the entire inheritance tree. One should note that this also works nicely as a mixin class. Trait descriptors ----------------- There are many designs of Python descriptors in the wild which, for example, check boundaries of values. Often those "traits" need some support of a metaclass to work. This is how this would look like with this PEP:: class Trait: def __get__(self, instance, owner): return instance.__dict__[self.key] def __set__(self, instance, value): instance.__dict__[self.key] = value def __set_owner__(self, owner, name): self.key = name Rejected Design Options ======================= Calling the hook on the class itself ------------------------------------ Adding an ``__autodecorate__`` hook that would be called on the class itself was the proposed idea of PEP 422. Most examples work the same way or even better if the hook is called on the subclass. In general, it is much easier to explicitly call the hook on the class in which it is defined (to opt-in to such a behavior) than to opt-out, meaning that one does not want the hook to be called on the class it is defined in. This becomes most evident if the class in question is designed as a mixin: it is very unlikely that the code of the mixin is to be executed for the mixin class itself, as it is not supposed to be a complete class on its own. The original proposal also made major changes in the class initialization process, rendering it impossible to back-port the proposal to older Python versions. More importantly, having a pure Python implementation allows us to take two preliminary steps before before we actually change the interpreter, giving us the chance to iron out all possible wrinkles in the API. Other variants of calling the hook ---------------------------------- Other names for the hook were presented, namely ``__decorate__`` or ``__autodecorate__``. This proposal opts for ``__init_subclass__`` as it is very close to the ``__init__`` method, just for the subclass, while it is not very close to decorators, as it does not return the class. Requiring an explicit decorator on ``__init_subclass__`` -------------------------------------------------------- One could require the explicit use of ``@classmethod`` on the ``__init_subclass__`` decorator. It was made implicit since there's no sensible interpretation for leaving it out, and that case would need to be detected anyway in order to give a useful error message. This decision was reinforced after noticing that the user experience of defining ``__prepare__`` and forgetting the ``@classmethod`` method decorator is singularly incomprehensible (particularly since PEP 3115 documents it as an ordinary method, and the current documentation doesn't explicitly say anything one way or the other). Defining arbitrary namespaces ----------------------------- PEP 422 defined a generic way to add arbitrary namespaces for class definitions. This approach is much more flexible than just leaving the definition order in a tuple. The ``__prepare__`` method in a metaclass supports exactly this behavior. But given that effectively the only use cases that could be found out in the wild were the ``OrderedDict`` way of determining the attribute order, it seemed reasonable to only support this special case. The metaclass described in this PEP has been designed to be very simple such that it could be reasonably made the default metaclass. This was especially important when designing the attribute order functionality: This was a highly demanded feature and has been enabled through the ``__prepare__`` method of metaclasses. This method can be abused in very weird ways, making it hard to correctly maintain this feature in CPython. This is why it has been proposed to deprecated this feature, and instead use ``OrderedDict`` as the standard namespace, supporting the most important feature while dropping most of the complexity. But this would have meant that ``OrderedDict`` becomes a language builtin like dict and set, and not just a standard library class. The choice of the ``__attribute_order__`` tuple is a much simpler solution to the problem. A more ``__new__``-like hook ---------------------------- In PEP 422 the hook worked more like the ``__new__`` method than the ``__init__`` method, meaning that it returned a class instead of modifying one. This allows a bit more flexibility, but at the cost of much harder implementation and undesired side effects. History ======= This used to be a competing proposal to PEP 422 by Nick Coghlan and Daniel Urban. PEP 422 intended to achieve the same goals as this PEP, but with a different way of implementation. In the meantime, PEP 422 has been withdrawn favouring this approach. References ========== .. _published code: http://mail.python.org/pipermail/python-dev/2012-June/119878.html .. _more than 10 years ago: http://mail.python.org/pipermail/python-dev/2001-November/018651.html .. _Zope's ExtensionClass: http://docs.zope.org/zope_secrets/extensionclass.html .. _uploaded to PyPI: https://pypi.python.org/pypi/metaclass Copyright ========= This document has been placed in the public domain.

10 19

Review of PEP 520: Ordered Class Definition Namespace
by Guido van Rossum 22 Jun '16

22 Jun '16

PEP 520 review notes. (From previous message; edited.) - I agree it's better to define the order as computed at runtime. - I don't think there's much of a point to mandate that all builtin/extension types reveal their order too -- I doubt there will be many uses for that -- but I don't want to disallow it either. We can allow types to define this, as long as it's in their documentation (so users can rely on it in those cases). - I don't like the exception for dunder names. I can see that __module__, __name__ etc. that occur in every class are distractions, but since you're adding methods, you should also add methods with dunder names like __init__ or __getattr__. (Otherwise, what if someone really wanted to create a Django form with a field named __dunder__?) - The overlap with PEP 487 makes me think that this feature is clearly desirable (I like the name you give it in PEP 520 better, and PEP 487 is too vague about its definition). - It seems someone is working on making all dicts ordered. Does that mean this will soon be obsolete? It would be a shame if we ended up having to give every class an extra attribute that is just a materialization of C.__dict__.keys() with (some) dunder names filtered out. (New) - It's a shame we can't just make __dict__ (a proxy to) an OrderedDict, then we wouldn't need an extra attribute. Hm, maybe we could customize the proxy class so its keys(), values(), items() views return things in the order of __definition_order__? (In the tracker discussion this was considered a big deal, but given that a class __dict__ is already a readonly proxy I'm not sure I agree. Or is this about C level access? How much of that would break?) - I don't see why it needs to be a read-only attribute. There are very few of those -- in general we let users play around with things unless we have a hard reason to restrict assignment (e.g. the interpreter's internal state could be compromised). I don't see such a hard reason here. - All in all the motivation is fairly weak -- it seems to be mostly motivated on avoiding a custom metaclass for this purpose because combining metaclasses is a pain. I realize it's only a small patch in a small corner of the language, but I do worry about repercussions -- it's an API that's going to be used for new (and useful) purposes so we will never be able to get rid of it. Note: I'm neither accepting nor rejecting the PEP; I'm merely inviting more contemplation. -- --Guido van Rossum (python.org/~guido)

3 3

frame evaluation API PEP
by Brett Cannon 21 Jun '16

21 Jun '16

For those of you who follow python-ideas or were at the PyCon US 2016 language summit, you have already seen/heard about this PEP. For those of you who don't fall into either of those categories, this PEP proposed a frame evaluation API for CPython. The motivating example of this work has been Pyjion, the experimental CPython JIT Dino Viehland and I have been working on in our spare time at Microsoft. The API also works for debugging, though, as already demonstrated by Google having added a very similar API internally for debugging purposes. The PEP is pasted in below and also available in rendered form at https://github.com/Microsoft/Pyjion/blob/master/pep.rst (I will assign myself a PEP # once discussion is finished as it's easier to work in git for this for the rich rendering of the in-progress PEP). I should mention that the difference from python-ideas and the language summit in the PEP are the listed support from Google's use of a very similar API as well as clarifying the co_extra field on code objects doesn't change their immutability (at least from the view of the PEP). ---------- PEP: NNN Title: Adding a frame evaluation API to CPython Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett(a)python.org>, Dino Viehland <dinov(a)microsoft.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 16-May-2016 Post-History: 16-May-2016 03-Jun-2016 Abstract ======== This PEP proposes to expand CPython's C API [#c-api]_ to allow for the specification of a per-interpreter function pointer to handle the evaluation of frames [#pyeval_evalframeex]_. This proposal also suggests adding a new field to code objects [#pycodeobject]_ to store arbitrary data for use by the frame evaluation function. Rationale ========= One place where flexibility has been lacking in Python is in the direct execution of Python code. While CPython's C API [#c-api]_ allows for constructing the data going into a frame object and then evaluating it via ``PyEval_EvalFrameEx()`` [#pyeval_evalframeex]_, control over the execution of Python code comes down to individual objects instead of a hollistic control of execution at the frame level. While wanting to have influence over frame evaluation may seem a bit too low-level, it does open the possibility for things such as a method-level JIT to be introduced into CPython without CPython itself having to provide one. By allowing external C code to control frame evaluation, a JIT can participate in the execution of Python code at the key point where evaluation occurs. This then allows for a JIT to conditionally recompile Python bytecode to machine code as desired while still allowing for executing regular CPython bytecode when running the JIT is not desired. This can be accomplished by allowing interpreters to specify what function to call to evaluate a frame. And by placing the API at the frame evaluation level it allows for a complete view of the execution environment of the code for the JIT. This ability to specify a frame evaluation function also allows for other use-cases beyond just opening CPython up to a JIT. For instance, it would not be difficult to implement a tracing or profiling function at the call level with this API. While CPython does provide the ability to set a tracing or profiling function at the Python level, this would be able to match the data collection of the profiler and quite possibly be faster for tracing by simply skipping per-line tracing support. It also opens up the possibility of debugging where the frame evaluation function only performs special debugging work when it detects it is about to execute a specific code object. In that instance the bytecode could be theoretically rewritten in-place to inject a breakpoint function call at the proper point for help in debugging while not having to do a heavy-handed approach as required by ``sys.settrace()``. To help facilitate these use-cases, we are also proposing the adding of a "scratch space" on code objects via a new field. This will allow per-code object data to be stored with the code object itself for easy retrieval by the frame evaluation function as necessary. The field itself will simply be a ``PyObject *`` type so that any data stored in the field will participate in normal object memory management. Proposal ======== All proposed C API changes below will not be part of the stable ABI. Expanding ``PyCodeObject`` -------------------------- One field is to be added to the ``PyCodeObject`` struct [#pycodeobject]_:: typedef struct { ... PyObject *co_extra; /* "Scratch space" for the code object. */ } PyCodeObject; The ``co_extra`` will be ``NULL`` by default and will not be used by CPython itself. Third-party code is free to use the field as desired. Values stored in the field are expected to not be required in order for the code object to function, allowing the loss of the data of the field to be acceptable (this keeps the code object as immutable from a functionality point-of-view; this is slightly contentious and so is listed as an open issue in `Is co_extra needed?`_). The field will be freed like all other fields on ``PyCodeObject`` during deallocation using ``Py_XDECREF()``. It is not recommended that multiple users attempt to use the ``co_extra`` simultaneously. While a dictionary could theoretically be set to the field and various users could use a key specific to the project, there is still the issue of key collisions as well as performance degradation from using a dictionary lookup on every frame evaluation. Users are expected to do a type check to make sure that the field has not been previously set by someone else. Expanding ``PyInterpreterState`` -------------------------------- The entrypoint for the frame evalution function is per-interpreter:: // Same type signature as PyEval_EvalFrameEx(). typedef PyObject* (__stdcall *PyFrameEvalFunction)(PyFrameObject*, int); typedef struct { ... PyFrameEvalFunction eval_frame; } PyInterpreterState; By default, the ``eval_frame`` field will be initialized to a function pointer that represents what ``PyEval_EvalFrameEx()`` currently is (called ``PyEval_EvalFrameDefault()``, discussed later in this PEP). Third-party code may then set their own frame evaluation function instead to control the execution of Python code. A pointer comparison can be used to detect if the field is set to ``PyEval_EvalFrameDefault()`` and thus has not been mutated yet. Changes to ``Python/ceval.c`` ----------------------------- ``PyEval_EvalFrameEx()`` [#pyeval_evalframeex]_ as it currently stands will be renamed to ``PyEval_EvalFrameDefault()``. The new ``PyEval_EvalFrameEx()`` will then become:: PyObject * PyEval_EvalFrameEx(PyFrameObject *frame, int throwflag) { PyThreadState *tstate = PyThreadState_GET(); return tstate->interp->eval_frame(frame, throwflag); } This allows third-party code to place themselves directly in the path of Python code execution while being backwards-compatible with code already using the pre-existing C API. Updating ``python-gdb.py`` -------------------------- The generated ``python-gdb.py`` file used for Python support in GDB makes some hard-coded assumptions about ``PyEval_EvalFrameEx()``, e.g. the names of local variables. It will need to be updated to work with the proposed changes. Performance impact ================== As this PEP is proposing an API to add pluggability, performance impact is considered only in the case where no third-party code has made any changes. Several runs of pybench [#pybench]_ consistently showed no performance cost from the API change alone. A run of the Python benchmark suite [#py-benchmarks]_ showed no measurable cost in performance. In terms of memory impact, since there are typically not many CPython interpreters executing in a single process that means the impact of ``co_extra`` being added to ``PyCodeObject`` is the only worry. According to [#code-object-count]_, a run of the Python test suite results in about 72,395 code objects being created. On a 64-bit CPU that would result in 579,160 bytes of extra memory being used if all code objects were alive at once and had nothing set in their ``co_extra`` fields. Example Usage ============= A JIT for CPython ----------------- Pyjion '''''' The Pyjion project [#pyjion]_ has used this proposed API to implement a JIT for CPython using the CoreCLR's JIT [#coreclr]_. Each code object has its ``co_extra`` field set to a ``PyjionJittedCode`` object which stores four pieces of information: 1. Execution count 2. A boolean representing whether a previous attempt to JIT failed 3. A function pointer to a trampoline (which can be type tracing or not) 4. A void pointer to any JIT-compiled machine code The frame evaluation function has (roughly) the following algorithm:: def eval_frame(frame, throw_flag): pyjion_code = frame.code.co_extra if not pyjion_code: frame.code.co_extra = PyjionJittedCode() elif not pyjion_code.jit_failed: if not pyjion_code.jit_code: return pyjion_code.eval(pyjion_code.jit_code, frame) elif pyjion_code.exec_count > 20_000: if jit_compile(frame): return pyjion_code.eval(pyjion_code.jit_code, frame) else: pyjion_code.jit_failed = True pyjion_code.exec_count += 1 return PyEval_EvalFrameDefault(frame, throw_flag) The key point, though, is that all of this work and logic is separate from CPython and yet with the proposed API changes it is able to provide a JIT that is compliant with Python semantics (as of this writing, performance is almost equivalent to CPython without the new API). This means there's nothing technically preventing others from implementing their own JITs for CPython by utilizing the proposed API. Other JITs '''''''''' It should be mentioned that the Pyston team was consulted on an earlier version of this PEP that was more JIT-specific and they were not interested in utilizing the changes proposed because they want control over memory layout they had no interest in directly supporting CPython itself. An informal discusion with a developer on the PyPy team led to a similar comment. Numba [#numba]_, on the other hand, suggested that they would be interested in the proposed change in a post-1.0 future for themselves [#numba-interest]_. The experimental Coconut JIT [#coconut]_ could have benefitted from this PEP. In private conversations with Coconut's creator we were told that our API was probably superior to the one they developed for Coconut to add JIT support to CPython. Debugging --------- In conversations with the Python Tools for Visual Studio team (PTVS) [#ptvs]_, they thought they would find these API changes useful for implementing more performant debugging. As mentioned in the Rationale_ section, this API would allow for switching on debugging functionality only in frames where it is needed. This could allow for either skipping information that ``sys.settrace()`` normally provides and even go as far as to dynamically rewrite bytecode prior to execution to inject e.g. breakpoints in the bytecode. It also turns out that Google has provided a very similar API internally for years. It has been used for performant debugging purposes. Implementation ============== A set of patches implementing the proposed API is available through the Pyjion project [#pyjion]_. In its current form it has more changes to CPython than just this proposed API, but that is for ease of development instead of strict requirements to accomplish its goals. Open Issues =========== Allow ``eval_frame`` to be ``NULL`` ----------------------------------- Currently the frame evaluation function is expected to always be set. It could very easily simply default to ``NULL`` instead which would signal to use ``PyEval_EvalFrameDefault()``. The current proposal of not special-casing the field seemed the most straight-forward, but it does require that the field not accidentally be cleared, else a crash may occur. Is co_extra needed? ------------------- While discussing this PEP at PyCon US 2016, some core developers expressed their worry of the ``co_extra`` field making code objects mutable. The thinking seemed to be that having a field that was mutated after the creation of the code object made the object seem mutable, even though no other aspect of code objects changed. The view of this PEP is that the `co_extra` field doesn't change the fact that code objects are immutable. The field is specified in this PEP as to not contain information required to make the code object usable, making it more of a caching field. It could be viewed as similar to the UTF-8 cache that string objects have internally; strings are still considered immutable even though they have a field that is conditionally set. The field is also not strictly necessary. While the field greatly simplifies attaching extra information to code objects, other options such as keeping a mapping of code object memory addresses to what would have been kept in ``co_extra`` or perhaps using a weak reference of the data on the code object and then iterating through the weak references until the attached data is found is possible. But obviously all of these solutions are not as simple or performant as adding the ``co_extra`` field. Rejected Ideas ============== A JIT-specific C API -------------------- Originally this PEP was going to propose a much larger API change which was more JIT-specific. After soliciting feedback from the Numba team [#numba]_, though, it became clear that the API was unnecessarily large. The realization was made that all that was truly needed was the opportunity to provide a trampoline function to handle execution of Python code that had been JIT-compiled and a way to attach that compiled machine code along with other critical data to the corresponding Python code object. Once it was shown that there was no loss in functionality or in performance while minimizing the API changes required, the proposal was changed to its current form. References ========== .. [#pyjion] Pyjion project (https://github.com/microsoft/pyjion) .. [#c-api] CPython's C API (https://docs.python.org/3/c-api/index.html) .. [#pycodeobject] ``PyCodeObject`` (https://docs.python.org/3/c-api/code.html#c.PyCodeObject) .. [#coreclr] .NET Core Runtime (CoreCLR) (https://github.com/dotnet/coreclr) .. [#pyeval_evalframeex] ``PyEval_EvalFrameEx()`` ( https://docs.python.org/3/c-api/veryhigh.html?highlight=pyframeobject#c.PyE… ) .. [#pycodeobject] ``PyCodeObject`` (https://docs.python.org/3/c-api/code.html#c.PyCodeObject) .. [#numba] Numba (http://numba.pydata.org/) .. [#numba-interest] numba-users mailing list: "Would the C API for a JIT entrypoint being proposed by Pyjion help out Numba?" ( https://groups.google.com/a/continuum.io/forum/#!topic/numba-users/yRl_0t8-… ) .. [#code-object-count] [Python-Dev] Opcode cache in ceval loop (https://mail.python.org/pipermail/python-dev/2016-February/143025.html) .. [#py-benchmarks] Python benchmark suite (https://hg.python.org/benchmarks) .. [#pyston] Pyston (http://pyston.org) .. [#pypy] PyPy (http://pypy.org/) .. [#ptvs] Python Tools for Visual Studio (http://microsoft.github.io/PTVS/) .. [#coconut] Coconut (https://github.com/davidmalcolm/coconut) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

7 15

PEP 520: Preserving Class Attribute Definition Order (round 4)
by Eric Snow 21 Jun '16

21 Jun '16

I've updated PEP 520 to reflect a clearer focus on the definition order and less emphasis on OrderedDict. -eric ======================================= PEP: 520 Title: Preserving Class Attribute Definition Order Version: $Revision$ Last-Modified: $Date$ Author: Eric Snow <ericsnowcurrently(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 7-Jun-2016 Python-Version: 3.6 Post-History: 7-Jun-2016, 11-Jun-2016, 20-Jun-2016 Abstract ======== When a class is defined using a ``class`` statement, the class body is executed within a namespace. After the execution completes, that namespace is copied into new ``dict`` and the original definition namespace is discarded. The new copy is stored away as the class's namespace and is exposed as ``__dict__`` through a read-only proxy. This PEP preserves the order in which the attributes in the definition namespace were added to it, before that namespace is discarded. This means it reflects the definition order of the class body. That order will now be preserved in the ``__definition_order__`` attribute of the class. This allows introspection of the original definition order, e.g. by class decorators. Additionally, this PEP changes the default class definition namespace to ``OrderedDict``. The long-lived class namespace (``__dict__``) will remain a ``dict``. Motivation ========== Currently Python does not preserve the order in which attributes are added to the class definition namespace. The namespace used during execution of a class body defaults to ``dict``. If the metaclass defines ``__prepare__()`` then the result of calling it is used. Thus, before this PEP, to access your class definition namespace you must use ``OrderedDict`` along with a metaclass. Then you must preserve the definition order (from the ``OrderedDict``) yourself. This has a couple of problems. First, it requires the use of a metaclass. Metaclasses introduce an extra level of complexity to code and in some cases (e.g. conflicts) are a problem. So reducing the need for them is worth doing when the opportunity presents itself. PEP 422 and PEP 487 discuss this at length. Given that we now have a C implementation of ``OrderedDict`` and that ``OrderedDict`` is the common use case for ``__prepare__()``, we have such an opportunity by defaulting to ``OrderedDict``. Second, only classes that opt in to using the ``OrderedDict``-based metaclass will have access to the definition order. This is problematic for cases where universal access to the definition order is important. One of the original motivating use cases for this PEP is generic class decorators that make use of the definition order. Specification ============= Part 1: * the order in which class attributes are defined is preserved in the new ``__definition_order__`` attribute on each class * "dunder" attributes (e.g. ``__init__``, ``__module__``) are ignored * ``__definition_order__`` is a ``tuple`` (or ``None``) * ``__definition_order__`` is a read-only attribute * ``__definition_order__`` is always set: 1. if ``__definition_order__`` is defined in the class body then it must be a ``tuple`` of identifiers or ``None``; any other value will result in ``TypeError`` 2. classes that do not have a class definition (e.g. builtins) have their ``__definition_order__`` set to ``None`` 3. classes for which `__prepare__()`` returned something other than ``OrderedDict`` (or a subclass) have their ``__definition_order__`` set to ``None`` (except where #1 applies) Part 2: * the default class *definition* namespace is now ``OrderdDict`` The following code demonstrates roughly equivalent semantics for the default behavior:: class Meta(type): @classmethod def __prepare__(cls, *args, **kwargs): return OrderedDict() class Spam(metaclass=Meta): ham = None eggs = 5 __definition_order__ = tuple(k for k in locals() if not (k.startswith('__') and k.endswith('__'))) Note that [pep487_] proposes a similar solution, albeit as part of a broader proposal. Why a tuple? ------------ Use of a tuple reflects the fact that we are exposing the order in which attributes on the class were *defined*. Since the definition is already complete by the time ``__definition_order__`` is set, the content and order of the value won't be changing. Thus we use a type that communicates that state of immutability. Why a read-only attribute? -------------------------- As with the use of tuple, making ``__definition_order__`` a read-only attribute communicates the fact that the information it represents is complete. Since it represents the state of a particular one-time event (execution of the class definition body), allowing the value to be replaced would reduce confidence that the attribute corresponds to the original class body. If a use case for a writable (or mutable) ``__definition_order__`` arises, the restriction may be loosened later. Presently this seems unlikely and furthermore it is usually best to go immutable-by-default. Note that ``__definition_order__`` is centered on the class definition body. The use cases for dealing with the class namespace (``__dict__``) post-definition are a separate matter. ``__definition_order__`` would be a significantly misleading name for a feature focused on more than class definition. See [nick_concern_] for more discussion. Why ignore "dunder" names? -------------------------- Names starting and ending with "__" are reserved for use by the interpreter. In practice they should not be relevant to the users of ``__definition_order__``. Instead, for nearly everyone they would only be clutter, causing the same extra work for everyone. Why None instead of an empty tuple? ----------------------------------- A key objective of adding ``__definition_order__`` is to preserve information in class definitions which was lost prior to this PEP. One consequence is that ``__definition_order__`` implies an original class definition. Using ``None`` allows us to clearly distinquish classes that do not have a definition order. An empty tuple clearly indicates a class that came from a definition statement but did not define any attributes there. Why None instead of not setting the attribute? ---------------------------------------------- The absence of an attribute requires more complex handling than ``None`` does for consumers of ``__definition_order__``. Why constrain manually set values? ---------------------------------- If ``__definition_order__`` is manually set in the class body then it will be used. We require it to be a tuple of identifiers (or ``None``) so that consumers of ``__definition_order__`` may have a consistent expectation for the value. That helps maximize the feature's usefulness. We could also also allow an arbitrary iterable for a manually set ``__definition_order__`` and convert it into a tuple. However, not all iterables infer a definition order (e.g. ``set``). So we opt in favor of requiring a tuple. Why is __definition_order__ even necessary? ------------------------------------------- Since the definition order is not preserved in ``__dict__``, it is lost once class definition execution completes. Classes *could* explicitly set the attribute as the last thing in the body. However, then independent decorators could only make use of classes that had done so. Instead, ``__definition_order__`` preserves this one bit of info from the class body so that it is universally available. Support for C-API Types ======================= Arguably, most C-defined Python types (e.g. built-in, extension modules) have a roughly equivalent concept of a definition order. So conceivably ``__definition_order__`` could be set for such types automatically. This PEP does not introduce any such support. However, it does not prohibit it either. Compatibility ============= This PEP does not break backward compatibility, except in the case that someone relies *strictly* on ``dict`` as the class definition namespace. This shouldn't be a problem since ``issubclass(OrderedDict, dict)`` is true. Changes ============= In addition to the class syntax, the following expose the new behavior: * builtins.__build_class__ * types.prepare_class * types.new_class Other Python Implementations ============================ Pending feedback, the impact on Python implementations is expected to be minimal. If a Python implementation cannot support switching to `OrderedDict``-by-default then it can always set ``__definition_order__`` to ``None``. Implementation ============== The implementation is found in the tracker. [impl_] Alternatives ============ cls.__dict__ as OrderedDict ------------------------------- Instead of storing the definition order in ``__definition_order__``, the now-ordered definition namespace could be copied into a new ``OrderedDict``. This would then be used as the mapping proxied as ``__dict__``. Doing so would mostly provide the same semantics. However, using ``OrderedDict`` for ``__dict__`` would obscure the relationship with the definition namespace, making it less useful. Additionally, doing this would require significant changes to the semantics of the concrete ``dict`` C-API. A "namespace" Keyword Arg for Class Definition ---------------------------------------------- PEP 422 introduced a new "namespace" keyword arg to class definitions that effectively replaces the need to ``__prepare__()``. [pep422_] However, the proposal was withdrawn in favor of the simpler PEP 487. A stdlib Metaclass that Implements __prepare__() with OrderedDict ----------------------------------------------------------------- This has all the same problems as writing your own metaclass. The only advantage is that you don't have to actually write this metaclass. So it doesn't offer any benefit in the context of this PEP. Set __definition_order__ at Compile-time ---------------------------------------- Each class's ``__qualname__`` is determined at compile-time. This same concept could be applied to ``__definition_order__``. The result of composing ``__definition_order__`` at compile-time would be nearly the same as doing so at run-time. Comparative implementation difficulty aside, the key difference would be that at compile-time it would not be practical to preserve definition order for attributes that are set dynamically in the class body (e.g. ``locals()[name] = value``). However, they should still be reflected in the definition order. One posible resolution would be to require class authors to manually set ``__definition_order__`` if they define any class attributes dynamically. Ultimately, the use of ``OrderedDict`` at run-time or compile-time discovery is almost entirely an implementation detail. References ========== .. [impl] issue #24254 (https://bugs.python.org/issue24254) .. [nick_concern] Nick's concerns about mutability (https://mail.python.org/pipermail/python-dev/2016-June/144883.html) .. [pep422] PEP 422 (https://www.python.org/dev/peps/pep-0422/#order-preserving-classes) .. [pep487] PEP 487 (https://www.python.org/dev/peps/pep-0487/#defining-arbitrary-namespaces) .. [orig] original discussion (https://mail.python.org/pipermail/python-ideas/2013-February/019690.html) .. [followup1] follow-up 1 (https://mail.python.org/pipermail/python-dev/2013-June/127103.html) .. [followup2] follow-up 2 (https://mail.python.org/pipermail/python-dev/2015-May/140137.html) Copyright =========== This document has been placed in the public domain.

1 0

New security-sig mailling list
by Ethan Furman 20 Jun '16

20 Jun '16

has been created: https://mail.python.org/mailman/listinfo/security-sig The purpose of this list is to discuss security-related enhancements to Python while having as little impact on backwards compatibility as possible. Once a proposal is ready it will be presented to Python Dev. (This text is subject to change. ;) -- ~Ethan~

2 2

Problem
by Sandranel 20 Jun '16

20 Jun '16

Hi: My daughter and I are trying to update to 8.1.2,but every time we try this happens >From the API To the Python window: Please advise

3 2

Discussion overload
by Kevin Ollivier 20 Jun '16

20 Jun '16

Hi all, Recent joiner here, I signed up after PyCon made me want to get more involved and have been lurking. I woke up this morning again to about 30 new messages in my inbox, almost all of which revolve around the os.urandom blocking discussion. There are just about hourly new posts showing up on this topic. There is such a thing as too much of a good thing. Discussion of issues is certainly good, but so far since joining this list I am seeing too much discussion happening too fast, and as someone who has been involved in open source for approaching two decades now, frankly, that is not really a good sign. The discussions are somewhat overlapping as so many people write back so quickly, there are multiple sub-discussions happening at once, and really at this point I'm not sure how much new each message is really adding, if anything at all. It seems to me the main solutions to this problem have all been identified, as have the tradeoffs of each. The discussion is now mostly at a point where people are just repeatedly debating (or promoting) the merits of their preferred solution and tradeoff. It is even spawning more abstract sub-discsussions about things like project compatibility policies. This discussion has really taken on a life of its own. For someone like me, a new joiner, seeing this makes me feel like wanting to simply unsubscribe. I've been on mailing lists where issues get debated endlessly, and at some point what inevitably happens is that the project starts to lose members who feel that even just trying to follow the discussions is eating up too much of their time. It really can suck the energy right out of a community. I don't want to see that happen to Python. I had a blast at PyCon, my first, and I really came away feeling more than ever that the community you have here is really special. The one problem I felt concerned about though, was that the core dev community risked a sense of paralysis caused by having too many cooks in the kitchen and too much worry about the potential unseen ramifications of changing things. That creates a sort of paralysis and difficulty achieving consensus on anything that, eventually, causes projects to slowly decline and be disrupted by a more agile alternative. Please consider taking a step back from this issue. Take a deep breath, and consider responding more slowly and letting people's points stew in your head for a day or two first. (Including this one pls. :) Python will not implode if you don't get that email out right away. If I understand what I've read of this torrent of messages correctly, we don't even know if there's a single real world use case where a user of os.urandom is hitting the same problem CPython did, so we don't even know if the blocking at startup issue is actually even happening in any real world Python code out there. It's clearly far from a rampant problem, in any case. Stop and think about that for a second. This is, in practice, potentially a complete non-issue. Fixing it in any number of ways may potentially change things for no one at all. You could even introduce a real problem while trying to fix a hypothetical one. There are more than enough real problems to deal with, so why push hypothetical problems to the top of your priority list? It's too easy to get caught up in the abstract nature of problems and to lose sight of the real people and code behind them, or sometimes, the lack thereof. Be practical, be pragmatic. Before you hit that reply button, think - in a practical sense, of all the things I could be doing right now, is this discussion the place where my involvement could generate the greatest positive impact for the project? Is this the biggest and most substantial problem the project should be focusing on right now? Projects and developers who know how to manage focus go on to achieve the greatest things, in my experience. Having been critical, I will end with a compliment. :) It is nice to see that with only a couple small exceptions, this discussion has remained very civil and respectful, which should be expected, but I know from experience that far too often these discussions start to take a nasty tone as people get frustrated. This is one of the things I really do love about the Python community, and it's one reason I want to see both the product and community grow and succeed even more. That, in fact, is why I'm choosing to write this message first rather than simply unsubscribe. Kevin

9 22