Mailman 3 June 2016 - Python-Dev

Issue 27417: Call CoInitializeEx on startup
by Steve Dower June 29, 2016

June 29, 2016

I know this is of fairly limited interest, so this is just advertising http://bugs.python.org/issue27417 where I propose enabling COM by default on startup. If you are someone who knows what CoInitializeEx is or why you may want to call it, I'm interested in your feedback/concerns. Come and post on http://bugs.python.org/issue27417. Cheers, Steve

1 0

Re: [Python-Dev] PEP 495 implementation
by Alexander Belopolsky June 28, 2016

June 28, 2016

Dear All, I have not received any responses since my first post in this thread in September last year. This time I am adding python-dev to BCC in hopes to reach a larger audience. With the date of the first beta (2016-09-07) fast approaching, I would like to commit PEP 495 implementation hopefully before alpha 3 (2016-07-11). I have a patch published as a pull request [1] against the python/cpython repository on github. If anyone still prefers the bug tracker workflow, I can publish it as a patch for issue #24773 [2] as well. Please also see my post from March below. Since that post, I have implemented a -utzdata option to the regression test, so now the long exhaustive test only runs when you do python -mtest -utzdata. Note that this test currently fails on a couple exotic timezones such as Asia/Riyadh87, but it is likely to be an issue with the timezone database rather than python code. I still don't have access to a Windows development box and I know that the current implementation will not work there because I use localtime_r. I need help/advise on this front. I have not started updating the documentation, so the PEP text [3] should serve as the documentation for now. I will try to get the documentation patch ready before the beta, but I don't want to delay checking in the code. On Mon, Mar 21, 2016 at 10:34 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > > Dear All, > > I am getting close to completing PEP 495 implementation, but I need someone to help me with a port to Windows. One of the major obstacles is that my implementation relies heavily on the POSIX localtime_r function which apparently is not available on Windows. > > I would also appreciate help from a unittest expert to improve test_datetime. One of the pressing tasks is to make ZoneInfoCompleteTest optional because it takes several minutes to complete. > > I am maintaining the patch as a pull request [1] against the python/cpython repository on github. Code reviews are most welcome. > > [1]: https://github.com/python/cpython/pull/20 [2]: http://bugs.python.org/issue24773 [3]: https://www.python.org/dev/peps/pep-0495

1 0

PEP 520: Preserving Class Attribute Definition Order (round 5)
by Eric Snow June 28, 2016

June 28, 2016

- a clearer motivation section - include "dunder" names - 2 open questions (__slots__? drop read-only requirement?) -eric ----------------------------------- PEP: 520 Title: Preserving Class Attribute Definition Order Version: $Revision$ Last-Modified: $Date$ Author: Eric Snow <ericsnowcurrently(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 7-Jun-2016 Python-Version: 3.6 Post-History: 7-Jun-2016, 11-Jun-2016, 20-Jun-2016, 24-Jun-2016 Abstract ======== The class definition syntax is ordered by its very nature. Class attributes defined there are thus ordered. Aside from helping with readability, that ordering is sometimes significant. If it were automatically available outside the class definition then the attribute order could be used without the need for extra boilerplate (such as metaclasses or manually enumerating the attribute order). Given that this information already exists, access to the definition order of attributes is a reasonable expectation. However, currently Python does not preserve the attribute order from the class definition. This PEP changes that by preserving the order in which attributes are introduced in the class definition body. That order will now be preserved in the ``__definition_order__`` attribute of the class. This allows introspection of the original definition order, e.g. by class decorators. Additionally, this PEP changes the default class definition namespace to ``OrderedDict``. The long-lived class namespace (``__dict__``) will remain a ``dict``. Motivation ========== The attribute order from a class definition may be useful to tools that rely on name order. However, without the automatic availability of the definition order, those tools must impose extra requirements on users. For example, use of such a tool may require that your class use a particular metaclass. Such requirements are often enough to discourage use of the tool. Some tools that could make use of this PEP include: * documentation generators * testing frameworks * CLI frameworks * web frameworks * config generators * data serializers * enum factories (my original motivation) Background ========== When a class is defined using a ``class`` statement, the class body is executed within a namespace. Currently that namespace defaults to ``dict``. If the metaclass defines ``__prepare__()`` then the result of calling it is used for the class definition namespace. After the execution completes, the definition namespace namespace is copied into new ``dict``. Then the original definition namespace is discarded. The new copy is stored away as the class's namespace and is exposed as ``__dict__`` through a read-only proxy. The class attribute definition order is represented by the insertion order of names in the *definition* namespace. Thus, we can have access to the definition order by switching the definition namespace to an ordered mapping, such as ``collections.OrderedDict``. This is feasible using a metaclass and ``__prepare__``, as described above. In fact, exactly this is by far the most common use case for using ``__prepare__`` (see PEP 487). At that point, the only missing thing for later access to the definition order is storing it on the class before the definition namespace is thrown away. Again, this may be done using a metaclass. However, this means that the definition order is preserved only for classes that use such a metaclass. There are two practical problems with that: First, it requires the use of a metaclass. Metaclasses introduce an extra level of complexity to code and in some cases (e.g. conflicts) are a problem. So reducing the need for them is worth doing when the opportunity presents itself. PEP 422 and PEP 487 discuss this at length. Given that we now have a C implementation of ``OrderedDict`` and that ``OrderedDict`` is the common use case for ``__prepare__()``, we have such an opportunity by defaulting to ``OrderedDict``. Second, only classes that opt in to using the ``OrderedDict``-based metaclass will have access to the definition order. This is problematic for cases where universal access to the definition order is important. Specification ============= Part 1: * all classes have a ``__definition_order__`` attribute * ``__definition_order__`` is a ``tuple`` of identifiers (or ``None``) * ``__definition_order__`` is a read-only attribute * ``__definition_order__`` is always set: 1. during execution of the class body, the insertion order of names into the class *definition* namespace is stored in a tuple 2. if ``__definition_order__`` is defined in the class body then it must be a ``tuple`` of identifiers or ``None``; any other value will result in ``TypeError`` 3. classes that do not have a class definition (e.g. builtins) have their ``__definition_order__`` set to ``None`` 4. classes for which `__prepare__()`` returned something other than ``OrderedDict`` (or a subclass) have their ``__definition_order__`` set to ``None`` (except where #2 applies) Part 2: * the default class *definition* namespace is now ``OrderdDict`` The following code demonstrates roughly equivalent semantics for the default behavior:: class Meta(type): @classmethod def __prepare__(cls, *args, **kwargs): return OrderedDict() class Spam(metaclass=Meta): ham = None eggs = 5 __definition_order__ = tuple(locals()) Why a tuple? ------------ Use of a tuple reflects the fact that we are exposing the order in which attributes on the class were *defined*. Since the definition is already complete by the time ``__definition_order__`` is set, the content and order of the value won't be changing. Thus we use a type that communicates that state of immutability. Why a read-only attribute? -------------------------- As with the use of tuple, making ``__definition_order__`` a read-only attribute communicates the fact that the information it represents is complete. Since it represents the state of a particular one-time event (execution of the class definition body), allowing the value to be replaced would reduce confidence that the attribute corresponds to the original class body. If a use case for a writable (or mutable) ``__definition_order__`` arises, the restriction may be loosened later. Presently this seems unlikely and furthermore it is usually best to go immutable-by-default. Note that the ability to set ``__definition_order__`` manually allows a dynamically created class (e.g. Cython, ``type()``) to still have ``__definition_order__`` properly set. Why not "__attribute_order__"? ------------------------------ ``__definition_order__`` is centered on the class definition body. The use cases for dealing with the class namespace (``__dict__``) post-definition are a separate matter. ``__definition_order__`` would be a significantly misleading name for a feature focused on more than class definition. Why not ignore "dunder" names? ------------------------------ Names starting and ending with "__" are reserved for use by the interpreter. In practice they should not be relevant to the users of ``__definition_order__``. Instead, for nearly everyone they would only be clutter, causing the same extra work for everyone. However, dropping dunder names by default may inadvertantly cause problems for classes that use dunder names unconventionally. In this case it's better to play it safe and preserve *all* the names from the class definition. Note that a couple of dunder names (``__name__`` and ``__qualname__``) are injected by default by the compiler. So they will be included even though they are not strictly part of the class definition body. Why None instead of an empty tuple? ----------------------------------- A key objective of adding ``__definition_order__`` is to preserve information in class definitions which was lost prior to this PEP. One consequence is that ``__definition_order__`` implies an original class definition. Using ``None`` allows us to clearly distinquish classes that do not have a definition order. An empty tuple clearly indicates a class that came from a definition statement but did not define any attributes there. Why None instead of not setting the attribute? ---------------------------------------------- The absence of an attribute requires more complex handling than ``None`` does for consumers of ``__definition_order__``. Why constrain manually set values? ---------------------------------- If ``__definition_order__`` is manually set in the class body then it will be used. We require it to be a tuple of identifiers (or ``None``) so that consumers of ``__definition_order__`` may have a consistent expectation for the value. That helps maximize the feature's usefulness. We could also also allow an arbitrary iterable for a manually set ``__definition_order__`` and convert it into a tuple. However, not all iterables infer a definition order (e.g. ``set``). So we opt in favor of requiring a tuple. Why is __definition_order__ even necessary? ------------------------------------------- Since the definition order is not preserved in ``__dict__``, it is lost once class definition execution completes. Classes *could* explicitly set the attribute as the last thing in the body. However, then independent decorators could only make use of classes that had done so. Instead, ``__definition_order__`` preserves this one bit of info from the class body so that it is universally available. Support for C-API Types ======================= Arguably, most C-defined Python types (e.g. built-in, extension modules) have a roughly equivalent concept of a definition order. So conceivably ``__definition_order__`` could be set for such types automatically. This PEP does not introduce any such support. However, it does not prohibit it either. The specific cases: * builtin types * PyType_Ready * PyType_FromSpec Compatibility ============= This PEP does not break backward compatibility, except in the case that someone relies *strictly* on ``dict`` as the class definition namespace. This shouldn't be a problem since ``issubclass(OrderedDict, dict)`` is true. Changes ============= In addition to the class syntax, the following expose the new behavior: * builtins.__build_class__ * types.prepare_class * types.new_class Other Python Implementations ============================ Pending feedback, the impact on Python implementations is expected to be minimal. If a Python implementation cannot support switching to `OrderedDict``-by-default then it can always set ``__definition_order__`` to ``None``. Open Questions ============== * What about `__slots__`? * Drop the "read-only attribute" requirement? Per Guido: I don't see why it needs to be a read-only attribute. There are very few of those -- in general we let users play around with things unless we have a hard reason to restrict assignment (e.g. the interpreter's internal state could be compromised). I don't see such a hard reason here. Implementation ============== The implementation is found in the tracker. [impl_] Alternatives ============ An Order-preserving cls.__dict__ -------------------------------- Instead of storing the definition order in ``__definition_order__``, the now-ordered definition namespace could be copied into a new ``OrderedDict``. This would then be used as the mapping proxied as ``__dict__``. Doing so would mostly provide the same semantics. However, using ``OrderedDict`` for ``__dict__`` would obscure the relationship with the definition namespace, making it less useful. Additionally, (in the case of ``OrderedDict`` specifically) doing this would require significant changes to the semantics of the concrete ``dict`` C-API. There has been some discussion about moving to a compact dict implementation which would (mostly) preserve insertion order. However the lack of an explicit ``__definition_order__`` would still remain as a pain point. A "namespace" Keyword Arg for Class Definition ---------------------------------------------- PEP 422 introduced a new "namespace" keyword arg to class definitions that effectively replaces the need to ``__prepare__()``. [pep422_] However, the proposal was withdrawn in favor of the simpler PEP 487. A stdlib Metaclass that Implements __prepare__() with OrderedDict ----------------------------------------------------------------- This has all the same problems as writing your own metaclass. The only advantage is that you don't have to actually write this metaclass. So it doesn't offer any benefit in the context of this PEP. Set __definition_order__ at Compile-time ---------------------------------------- Each class's ``__qualname__`` is determined at compile-time. This same concept could be applied to ``__definition_order__``. The result of composing ``__definition_order__`` at compile-time would be nearly the same as doing so at run-time. Comparative implementation difficulty aside, the key difference would be that at compile-time it would not be practical to preserve definition order for attributes that are set dynamically in the class body (e.g. ``locals()[name] = value``). However, they should still be reflected in the definition order. One posible resolution would be to require class authors to manually set ``__definition_order__`` if they define any class attributes dynamically. Ultimately, the use of ``OrderedDict`` at run-time or compile-time discovery is almost entirely an implementation detail. References ========== .. [impl] issue #24254 (https://bugs.python.org/issue24254) .. [nick_concern] Nick's concerns about mutability (https://mail.python.org/pipermail/python-dev/2016-June/144883.html) .. [pep422] PEP 422 (https://www.python.org/dev/peps/pep-0422/#order-preserving-classes) .. [pep487] PEP 487 (https://www.python.org/dev/peps/pep-0487/#defining-arbitrary-namespaces) .. [orig] original discussion (https://mail.python.org/pipermail/python-ideas/2013-February/019690.html) .. [followup1] follow-up 1 (https://mail.python.org/pipermail/python-dev/2013-June/127103.html) .. [followup2] follow-up 2 (https://mail.python.org/pipermail/python-dev/2015-May/140137.html) Copyright =========== This document has been placed in the public domain.

6 12

[RELEASE] Python 2.7.12
by Benjamin Peterson June 28, 2016

June 28, 2016

It is my privilege to present you with another release in the Python 2.7 series, Python 2.7.12. Since the release candidate, there were two changes: - The Windows binaries have been changed to use OpenSSL 1.0.2h. - The "about" dialog in IDLE was fixed. Downloads, as always, are on python.org: https://www.python.org/downloads/release/python-2712/ The complete 2.7.12 changelog is available at https://hg.python.org/cpython/raw-file/v2.7.12/Misc/NEWS Yet another Python 2.7.x release is anticipated near the end of the year. Numerologists may wish to upgrade to Python 3 before we hit the unlucky 2.7.13. Servus, Benjamin 2.7 release manager

1 0

When to use EOFError?
by Serhiy Storchaka June 27, 2016

June 27, 2016

There is a design question. If you read file in some format or with some protocol, and the data is ended unexpectedly, when to use general EOFError exception and when to use format/protocol specific exception? For example when load truncated pickle data, an unpickler can raise EOFError, UnpicklingError, ValueError or AttributeError. It is possible to avoid ValueError or AttributeError, but what exception should be raised instead, EOFError or UnpicklingError? Maybe convert all EOFError to UnpicklingError? Or all UnpicklingError caused by unexpectedly ended input to EOFError? Or raise EOFError if the input is ended after completed opcode, and UnpicklingError if it contains truncated opcode? http://bugs.python.org/issue25761

11 17

[RELEASED] Python 3.4.5 and Python 3.5.2 are now available
by Larry Hastings June 27, 2016

June 27, 2016

On behalf of the Python development community and the Python 3.4 and Python 3.5 release teams, I'm thrilled to announce the availability of Python 3.4.5 and Python 3.5.2. Python 3.4 is now in "security fixes only" mode. This is the final stage of support for Python 3.4. All changes made to Python 3.4 since Python 3.4.4 should be security fixes only; conventional bug fixes are not accepted. Also, Python 3.4.5 and all future releases of Python 3.4 will only be released as source code--no official binary installers will be produced. Python 3.5 is still in active "bug fix" mode. Python 3.5.2 contains many incremental improvements and bug fixes over Python 3.5.1. You can find Python 3.4.5 here: https://www.python.org/downloads/release/python-345/ And you can find Python 3.5.2 here: https://www.python.org/downloads/release/python-352/ Releasing software from 30,000 feet, //arry /p.s. There appears to be a small oops with the Windows installers for 3.5.2--uploaded to the wrong directory or something. They'll be available soon, honest!

2 1

Compact ordered dict is not ordered for split table. (was: PEP XXX: Compact ordered dict
by INADA Naoki June 26, 2016

June 26, 2016

I'm sorry, but I hadn't realized which compact ordered dict is not ordered for split table. For example: >>> class A: ... ... ... >>> a = A() >>> b = A() >>> a.a = 1 >>> a.b = 2 >>> b.b = 3 >>> b.a = 4 >>> a.__dict__.items() dict_items([('a', 1), ('b', 2)]) >>> b.__dict__.items() dict_items([('a', 4), ('b', 3)]) This doesn't affects to **kwargs and class namespace. But if we change the language spec to dict preserves insertion order, this should be addressed. On Tue, Jun 21, 2016 at 2:02 PM, INADA Naoki <songofacandy(a)gmail.com> wrote: > On Tue, Jun 21, 2016 at 12:17 PM, Oleg Broytman <phd(a)phdru.name> wrote: >> Hi! >> >> On Tue, Jun 21, 2016 at 11:14:39AM +0900, INADA Naoki <songofacandy(a)gmail.com> wrote: >>> Here is my draft, but I haven't >>> posted it yet since >>> my English is much worse than C. >>> https://www.dropbox.com/s/s85n9b2309k03cq/pep-compact-dict.txt?dl=0 >> >> It's good enough for a start (if a PEP is needed at all). If you push >> it to Github I'm sure they will come with pull requests. >> >> Oleg. > > Thank you for reading my draft. > >> (if a PEP is needed at all) > > I don't think so. My PEP is not for changing Python Language, > just describe implementation detail. > > Python 3.5 has new OrderedDict implemented in C without PEP. > My patch is relatively small than it. And the idea has been well known. > > -- > INADA Naoki <songofacandy(a)gmail.com> -- INADA Naoki <songofacandy(a)gmail.com>

3 6

PEP 520: Ordered Class Definition Namespace
by Eric Snow June 24, 2016

June 24, 2016

I've grabbed a PEP # (520) and updated the PEP to clarify points that were brought up earlier today. Given positive feedback I got at PyCon and the reaction today, I'm hopeful the PEP isn't far off from pronouncement. :) -eric ========================================== PEP: 520 Title: Ordered Class Definition Namespace Version: $Revision$ Last-Modified: $Date$ Author: Eric Snow <ericsnowcurrently(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 7-Jun-2016 Python-Version: 3.6 Post-History: 7-Jun-2016 Abstract ======== This PEP changes the default class definition namespace to ``OrderedDict``. Furthermore, the order in which the attributes are defined in each class body will now be preserved in ``type.__definition_order__``. This allows introspection of the original definition order, e.g. by class decorators. Note: just to be clear, this PEP is *not* about changing ``__dict__`` for classes to ``OrderedDict``. Motivation ========== Currently the namespace used during execution of a class body defaults to ``dict``. If the metaclass defines ``__prepare__()`` then the result of calling it is used. Thus, before this PEP, if you needed your class definition namespace to be ``OrderedDict`` you had to use a metaclass. Metaclasses introduce an extra level of complexity to code and in some cases (e.g. conflicts) are a problem. So reducing the need for them is worth doing when the opportunity presents itself. Given that we now have a C implementation of ``OrderedDict`` and that ``OrderedDict`` is the common use case for ``__prepare__()``, we have such an opportunity by defaulting to ``OrderedDict``. The usefulness of ``OrderedDict``-by-default is greatly increased if the definition order is directly introspectable on classes afterward, particularly by code that is independent of the original class definition. One of the original motivating use cases for this PEP is generic class decorators that make use of the definition order. Changing the default class definition namespace has been discussed a number of times, including on the mailing lists and in PEP 422 and PEP 487 (see the References section below). Specification ============= * the default class *definition* namespace is now ``OrderdDict`` * the order in which class attributes are defined is preserved in the new ``__definition_order__`` attribute on each class * "dunder" attributes (e.g. ``__init__``, ``__module__``) are ignored * ``__definition_order__`` is a tuple * ``__definition_order__`` is a read-only attribute * ``__definition_order__`` is always set: 1. if ``__definition_order__`` is defined in the class body then the value is used as-is, though the attribute will still be read-only 2. types that do not have a class definition (e.g. builtins) have their ``__definition_order__`` set to ``None`` 3. types for which `__prepare__()`` returned something other than ``OrderedDict`` (or a subclass) have their ``__definition_order__`` set to ``None`` (except where #1 applies) The following code demonstrates roughly equivalent semantics:: class Meta(type): def __prepare__(cls, *args, **kwargs): return OrderedDict() class Spam(metaclass=Meta): ham = None eggs = 5 __definition_order__ = tuple(k for k in locals() if (!k.startswith('__') or !k.endswith('__'))) Note that [pep487_] proposes a similar solution, albeit as part of a broader proposal. Why a tuple? ------------ Use of a tuple reflects the fact that we are exposing the order in which attributes on the class were *defined*. Since the definition is already complete by the time ``definition_order__`` is set, the content and order of the value won't be changing. Thus we use a type that communicates that state of immutability. Why a read-only attribute? -------------------------- As with the use of tuple, making ``__definition_order__`` a read-only attribute communicates the fact that the information it represents is complete. Since it represents the state of a particular one-time event (execution of the class definition body), allowing the value to be replaced would reduce confidence that the attribute corresponds to the original class body. If a use case for a writable (or mutable) ``__definition_order__`` arises, the restriction may be loosened later. Presently this seems unlikely and furthermore it is usually best to go immutable-by-default. Note that ``__definition_order__`` is centered on the class definition body. The use cases for dealing with the class namespace (``__dict__``) post-definition are a separate matter. ``__definition_order__`` would be a significantly misleading name for a supporting feature. See [nick_concern_] for more discussion. Why ignore "dunder" names? -------------------------- Names starting and ending with "__" are reserved for use by the interpreter. In practice they should not be relevant to the users of ``__definition_order__``. Instead, for early everyone they would only be clutter, causing the same extra work for everyone. Why is __definition_order__ even necessary? ------------------------------------------- Since the definition order is not preserved in ``__dict__``, it would be lost once class definition execution completes. Classes *could* explicitly set the attribute as the last thing in the body. However, then independent decorators could only make use of classes that had done so. Instead, ``__definition_order__`` preserves this one bit of info from the class body so that it is universally available. Compatibility ============= This PEP does not break backward compatibility, except in the case that someone relies *strictly* on ``dict`` as the class definition namespace. This shouldn't be a problem. Changes ============= In addition to the class syntax, the following expose the new behavior: * builtins.__build_class__ * types.prepare_class * types.new_class Other Python Implementations ============================ Pending feedback, the impact on Python implementations is expected to be minimal. If a Python implementation cannot support switching to `OrderedDict``-by-default then it can always set ``__definition_order__`` to ``None``. Implementation ============== The implementation is found in the tracker. [impl_] Alternatives ============ type.__dict__ as OrderedDict ---------------------------- Instead of storing the definition order in ``__definition_order__``, the now-ordered definition namespace could be copied into a new ``OrderedDict``. This would mostly provide the same semantics. However, using ``OrderedDict`` for ``type,__dict__`` would obscure the relationship with the definition namespace, making it less useful. Additionally, doing this would require significant changes to the semantics of the concrete ``dict`` C-API. A "namespace" Keyword Arg for Class Definition ---------------------------------------------- PEP 422 introduced a new "namespace" keyword arg to class definitions that effectively replaces the need to ``__prepare__()``. [pep422_] However, the proposal was withdrawn in favor of the simpler PEP 487. References ========== .. [impl] issue #24254 (https://bugs.python.org/issue24254) .. [nick_concern] Nick's concerns about mutability (https://mail.python.org/pipermail/python-dev/2016-June/144883.html) .. [pep422] PEP 422 (https://www.python.org/dev/peps/pep-0422/#order-preserving-classes) .. [pep487] PEP 487 (https://www.python.org/dev/peps/pep-0487/#defining-arbitrary-namespaces) .. [orig] original discussion (https://mail.python.org/pipermail/python-ideas/2013-February/019690.html) .. [followup1] follow-up 1 (https://mail.python.org/pipermail/python-dev/2013-June/127103.html) .. [followup2] follow-up 2 (https://mail.python.org/pipermail/python-dev/2015-May/140137.html) Copyright =========== This document has been placed in the public domain.

10 26

unittest.TestResult lacks API to separate subtests
by Ilya Kazakevich June 24, 2016

June 24, 2016

Hello, We're developing Python IDE and integrated it with unittest module using TestResult <https://docs.python.org/3.4/library/unittest.html#unittest.TestResult> inheritor to track test start, end etc. With Py3K, it supports addSubTest method, that is called after all subtests. But there is no method called before and after _each_ subtest (like it is done for regular tests). With out of it I can't fetch each subtest output and display it correctly. I suggest to add subTestStart / subTestEnd methods to help me with my issue and other people with similar issues. I can send patch if you think this is a good idea. Ilya Kazakevich JetBrains <http://www.jetbrains.com/> http://www.jetbrains.com The Drive to Develop

2 1

Idea: more compact, interned string key only dict for namespace.
by INADA Naoki June 24, 2016

June 24, 2016

As my last email, compact ordered dict can't preserve insertion order of key sharing dict (PEP 412). I'm thinking about deprecating key shared dict for now. Instead, my new idea is introducing more compact dict specialized for namespace. If BDFL (or BDFL delegate) likes this idea, I'll take another one week to implement this. Background ---------------- * Most keys of namespace dict are string. * Calculating hash of string is cheap (one memory access, thanks for cache). * And most keys are interned already. Design ---------- Instead of normal PyDictKeyEntry, use PyInternedKeyEntry like this. typedef struct { // no me_hash PyObject *me_key, *me_value; } PyInternedKeyEntry; insertdict() interns key if it's unicode, otherwise it converts dict to normal compact ordered dict. lookdict_interned() compares only pointer (doesn't call unicode_eq()) when searching key is interned. And add new internal API to create interned key only dict. PyDictObject* _PyDict_NewForNamespace(); Memory usage -------------------- on amd64 arch. key-sharing dict: * 96 bytes for ~3 items * 128 bytes for 4~5 items. compact dict: * 224 bytes for ~5 items. (232 bytes when keep supporting key-shared dict) interned key only dict: * 184 bytes for ~5 items Note ------ Interned key only dict is still larger than key-shared dict. But it can be used for more purpose. It can be used for interning string for example. It can be used to kwargs dict when all keys are interned already. If we provide _PyDict_NewForNamespace to extension modules, json decoder can have option to use this, too. -- INADA Naoki <songofacandy(a)gmail.com>

2 5