Mailman 3 September 2017 - Python-Dev

Reminder: snapshots and releases coming up in the next several days
by Ned Deily 13 Sep '17

13 Sep '17

Lots of releases coming up soon! 2017-09-16: - Python 2.7.14 final release 2017-09-18 1200 UTC cutoff: - Python 3.6.3 rc1 - Python 3.3.7 final release (following by retirement) If you know of any issues that you feel need to be addressed in these releases, please make sure there is an open issue on the bug tracker set to "release blocker". Also on 2017-09-18: - Python 3.7.0 alpha 1 This will be the first preview snapshot of 3.7.0. It's still very early in the development cycle for 3.7. The main purpose of early alpha releases is to exercise the build and release process and to make it easier for Windows and Macs users to help test. Many new features and build changes are yet to appear in subsequent releases prior to feature code cutoff on 2018-01-29 at 3.7.0b1. Thanks for your help! --Ned -- Ned Deily nad(a)python.org -- []

2 2

PEP 490 "Chain exceptions at C level" rejected
by Victor Stinner 12 Sep '17

12 Sep '17

Hi, In March 2015, I proposed the PEP 490 to chain implicitly C exceptions by default: https://www.python.org/dev/peps/pep-0490/ The discussion on python-dev was quickly in favor of keeping the status quo: don't chain C exceptions by default, but only do that explicitly where it makes sense. I forgot this PEP, but I just rejected it to close the old discussion :-) Note: The PEP is not yet rejected on python.org, it will be done at the next cron job run. Victor

3 2

breakpoint() and $PYTHONBREAKPOINT
by Barry Warsaw 12 Sep '17

12 Sep '17

For PEP 553, I think it’s a good idea to support the environment variable $PYTHONBREAKPOINT[*] but I’m stuck on a design question, so I’d like to get some feedback. Should $PYTHONBREAKPOINT be consulted in breakpoint() or in sys.breakpointhook()? If we support it in breakpoint() then it means if $PYTHONBREAKPOINT is set, Python will never call sys.breakpointhook() so programmatic override of that will be ignored. If we support it in sys.breakpointhook(), then if you programmatically override that, $PYTHONBREAKPOINT will be ignored even if set. Of course, any code that overrides sys.breakpointhook() could also consult $PYTHONBREAKPOINT if they want. So how do we want it to work? It seems like explicit code should override the environment variable, which should override the default, meaning that $PYTHONBREAKPOINT should be consulted only in breakpoint(). And if you set sys.breakpointhook() with something else, then oh well, Python just ignores $PYTHONBREAKPOINT. Feedback welcome. -Barry [*] Probably not $PYTHONBREAKPOINTHOOK although if we choose to implement that in sys.breakpointhook(), it might still makes sense.

4 8

Aaron Hall, Introduction and pull-request bump
by Aaron Hall 12 Sep '17

12 Sep '17

Hi there, I have had the privilege of getting to know some core developers at PyCon in Portland, and I've met some others through my volunteering at PyGotham and the NYC Python meetup group (resisting urge to name-drop). In spite of not joining the mailing list until about a week ago at Brett Cannon's suggestion, I have managed to submit 3 bug-report fixes, with 1 closed by pull-request, 1 nearly closed with the pull-request accepted, and 1 with the pull request not yet accepted (looks like there's a conflict in misc/NEWS now, not sure how to use Github to resolve it either). That last pull request is here: bpo-26103 resolve Data descriptor contradiction by aaronchall · Pull Request #1959 · python/cpython - probably good if | | | bpo-26103 resolve Data descriptor contr... | I also had a "MutableNamedTuple" implementation I put on codereview.stackexchange.com (https://codereview.stackexchange.com/q/173045/23451) and Ashwini Chaudhury gave me some excellent pointers, and inspired by the recent discussion on "Data Classes" I now have a half-baked implementation that leverages __slots__ with a Mapping, for example (from a unittest, hence the indentation): class MNTDemo(MutableNamedTuple): "Demo" __slots__ = { 'arg0': arg(), 'arg1': arg(typing.List), 'kwarg': arg(default=0), # must be positional if passed *args 'args': arg(stars=Stars.ARGS), # everything after *args must be keyword argument 'kwarg1': arg(list, []), 'kwarg2': None, 'kwargs': arg(typing.Any, stars=Stars.KWARGS) } mnt = MNTDemo(1, 2, 3, 'stararg1', kwarg1=1, kwarg2=2, kwarg3=3) Maybe there's a nicer way to do it, but arg is a function returning a namedtuple (attrs: type, default, stars) with some defaults. It allows the __init__ to have required positional arguments and required named arguments, pretty much as flexible as a regular function signature, but communicated via __slots__. And thanks to the slots, they're every bit as performant as a namedtuple, I presume (I gave the __slots__ talk at PyCon). Anyways, to wrap up my introduction, I'm a moderator on StackOverflow (https://stackoverflow.com/users/541136/aaron-hall?tab=profile) and nearly about to catch Raymond in reputation points, mostly due to Python, I'm currently teaching Python at NYU (Introduction to Programming), and I'm an all-around Python evangelist (as much as I know how to be.) I owe a lot to the Python community, especially the meetup community in NYC, but also virtual community (Ned Batchelder in IRC comes to mind). Thank you for everything, I'm looking for my chance to give back! Cheers, Aaron Hall

2 1

Python vulnerabilities
by Stephen Michell 11 Sep '17

11 Sep '17

I am new to this list. Skip suggested that I join. I convene ISO/IEC/JTC1SC22/WG23 Programming Languages Working Group. We produce a suite of international technical reports that document vulnerabilities in programming that can lead to serious safety and security breaches. We published TR 24772 "Guidance to avoiding programming language vulnerabilities through language selection and use" in 2010 and again in 2013. Edition one was a language independent look at such vulnerabilities. Edition two added new vulnerabilities plus language specific annexes for Ada, C, Python, PHP, Ruby, and Spark. For this round, we have split the document into parts and are publishing the language specific parts separately. We have added a few new vulnerabilities, mostly associated with concurrency and object orientation for this iteration. We target the team lead that guides and writes coding standards for an organization, as opposed to the general programmer. We plan to ballot and publish in 2018 TR 24772-1, the language independent Part, as well as -2 Ada, -3 C, -4 Python and -8 Fortran. Our Python Part needs completion to address the new vulnerabilities documented. We want to do justice to all languages that we work with. We need experts to help us complete the document, and then to review it. I have had initial conversations with one expert. We hope for a bit more if possible. I If interested, please contact me as listed below. Our document list is at www.open-std.org/JTC1/sc22/wg23. Thank you. Stephen Michell Maurya Software stephen dot michell at maurya dot on dot ca Phone: 1-613-299-9047

1 0

Fwd: Python programming language vulnerabilities
by Skip Montanaro 11 Sep '17

11 Sep '17

This popped up on python-list. It actually seems to me like it might be interesting to the core developers. Apologies if I've missed my guess. Skip ---------- Forwarded message ---------- From: Stephen Michell <Stephen.Michell(a)maurya.on.ca> Date: Fri, Sep 8, 2017 at 12:34 PM Subject: Python programming language vulnerabilities To: python-list(a)python.org I chair ISO/IEC/JTC1/SC22/WG23 Programming Language Vulnerabilities. We publish an international technical report, ISO IEC TR 24772 Guide to avoiding programming language vulnerabilities through language selection use. Annex D in this document addresses vulnerabilities in Python. This document is freely available from ISO and IEC. We are updating this technical report, adding a few vulnerabilities and updating language applicability as programming languages evolve. We are also subdividing the document by making the language-specific annexes each their own technical report. For the Python Part, the major portions are written, but we have about 6 potential vulnerabilities left to complete. We need help in finishing the Python TR. We are looking for a few Python experts that have experience in implementing Python language systems, or experts in implementing significant systems in Python (for technical level, persons that provide technical supervision to implementers, or that write and maintain organizational Python coding standards. If you are interested in helping, please reply to this posting. Thank you Stephen Michell Convenor, ISO/IEC/JTC 1/SC 22/WG 23 Programming Language Vulnerabilities -- https://mail.python.org/mailman/listinfo/python-list

3 2

PEP 556: Threaded garbage collection
by Antoine Pitrou 09 Sep '17

09 Sep '17

Hello, I've written a PEP by which you can tell the GC to run in a dedicated thread. The goal is to solve reentrancy issues with finalizers: https://www.python.org/dev/peps/pep-0556/ Regards Antoine. PS: I did not come up with the idea for this PEP while other people were having nightmares. Any nightmares involved in this PEP are fictional, and any resemblance to actual nightmares is purely coincidental. No nightmares were harmed while writing this PEP.

6 12

PEP 539 v3: A new C API for Thread-Local Storage in CPython
by Masayuki YAMAMOTO 09 Sep '17

09 Sep '17

Hi folks, I submit PEP 539 third draft for the finish. Thank you for all the advice and the help! Summary of technical changes: * Don't use `bool` types in the implementation, per discussion at python/cpython#1362 [1] * Update for removal of --without-threads Best regards, Masayuki [1]: https://github.com/python/cpython/pull/1362#discussion_r136357583 First round: https://mail.python.org/pipermail/python-ideas/2016-December/043983.html Second round: https://mail.python.org/pipermail/python-dev/2017-August/149091.html Discussion for the issue: https://bugs.python.org/issue25658 HTML version for PEP 539 draft: https://www.python.org/dev/peps/pep-0539/ Diff from first round to version 3: https://gist.github.com/ma8ma/624f9e4435ebdb26230130b11ce12d20/revisions And the pull-request for reference implementation (work in progress): https://github.com/python/cpython/pull/1362 ======================================== PEP: 539 Title: A New C-API for Thread-Local Storage in CPython Version: $Revision$ Last-Modified: $Date$ Author: Erik M. Bray, Masayuki Yamamoto BDFL-Delegate: Nick Coghlan Status: Draft Type: Informational Content-Type: text/x-rst Created: 20-Dec-2016 Post-History: 16-Dec-2016 Abstract ======== The proposal is to add a new Thread Local Storage (TLS) API to CPython which would supersede use of the existing TLS API within the CPython interpreter, while deprecating the existing API. The new API is named the "Thread Specific Storage (TSS) API" (see `Rationale for Proposed Solution`_ for the origin of the name). Because the existing TLS API is only used internally (it is not mentioned in the documentation, and the header that defines it, ``pythread.h``, is not included in ``Python.h`` either directly or indirectly), this proposal probably only affects CPython, but might also affect other interpreter implementations (PyPy?) that implement parts of the CPython API. This is motivated primarily by the fact that the old API uses ``int`` to represent TLS keys across all platforms, which is neither POSIX-compliant, nor portable in any practical sense [1]_. .. note:: Throughout this document the acronym "TLS" refers to Thread Local Storage and should not be confused with "Transportation Layer Security" protocols. Specification ============= The current API for TLS used inside the CPython interpreter consists of 6 functions:: PyAPI_FUNC(int) PyThread_create_key(void) PyAPI_FUNC(void) PyThread_delete_key(int key) PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value) PyAPI_FUNC(void *) PyThread_get_key_value(int key) PyAPI_FUNC(void) PyThread_delete_key_value(int key) PyAPI_FUNC(void) PyThread_ReInitTLS(void) These would be superseded by a new set of analogous functions:: PyAPI_FUNC(int) PyThread_tss_create(Py_tss_t *key) PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t *key) PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t *key, void *value) PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t *key) The specification also adds a few new features: * A new type ``Py_tss_t``--an opaque type the definition of which may depend on the underlying TLS implementation. It is defined:: typedef struct { int _is_initialized; NATIVE_TSS_KEY_T _key; } Py_tss_t; where ``NATIVE_TSS_KEY_T`` is a macro whose value depends on the underlying native TLS implementation (e.g. ``pthread_key_t``). * A constant default value for ``Py_tss_t`` variables, ``Py_tss_NEEDS_INIT``. * Three new functions:: PyAPI_FUNC(Py_tss_t *) PyThread_tss_alloc(void) PyAPI_FUNC(void) PyThread_tss_free(Py_tss_t *key) PyAPI_FUNC(int) PyThread_tss_is_created(Py_tss_t *key) The first two are needed for dynamic (de-)allocation of a ``Py_tss_t``, particularly in extension modules built with ``Py_LIMITED_API``, where static allocation of this type is not possible due to its implementation being opaque at build time. A value returned by ``PyThread_tss_alloc`` is in the same state as a value initialized with ``Py_tss_NEEDS_INIT``, or ``NULL`` in the case of dynamic allocation failure. The behavior of ``PyThread_tss_free`` involves calling ``PyThread_tss_delete`` preventively, or is a no-op if the value pointed to by the ``key`` argument is ``NULL``. ``PyThread_tss_is_created`` returns non-zero if the given ``Py_tss_t`` has been initialized (i.e. by ``PyThread_tss_create``). The new TSS API does not provide functions which correspond to ``PyThread_delete_key_value`` and ``PyThread_ReInitTLS``, because these functions were needed only for CPython's now defunct built-in TLS implementation; that is the existing behavior of these functions is treated as follows: ``PyThread_delete_key_value(key)`` is equalivalent to ``PyThread_set_key_value(key, NULL)``, and ``PyThread_ReInitTLS()`` is a no-op [8]_. The new ``PyThread_tss_`` functions are almost exactly analogous to their original counterparts with a few minor differences: Whereas ``PyThread_create_key`` takes no arguments and returns a TLS key as an ``int``, ``PyThread_tss_create`` takes a ``Py_tss_t*`` as an argument and returns an ``int`` status code. The behavior of ``PyThread_tss_create`` is undefined if the value pointed to by the ``key`` argument is not initialized by ``Py_tss_NEEDS_INIT``. The returned status code is zero on success and non-zero on failure. The meanings of non-zero status codes are not otherwise defined by this specification. Similarly the other ``PyThread_tss_`` functions are passed a ``Py_tss_t*`` whereas previously the key was passed by value. This change is necessary, as being an opaque type, the ``Py_tss_t`` type could hypothetically be almost any size. This is especially necessary for extension modules built with ``Py_LIMITED_API``, where the size of the type is not known. Except for ``PyThread_tss_free``, the behaviors of ``PyThread_tss_`` are undefined if the value pointed to by the ``key`` argument is ``NULL``. Moreover, because of the use of ``Py_tss_t`` instead of ``int``, there are behaviors in the new API which differ from the existing API with regard to key creation and deletion. ``PyThread_tss_create`` can be called repeatedly on the same key--calling it on an already initialized key is a no-op and immediately returns success. Similarly for calling ``PyThread_tss_delete`` with an uninitialized key. The behavior of ``PyThread_tss_delete`` is defined to change the key's initialization state to "uninitialized"--this allows, for example, statically allocated keys to be reset to a sensible state when restarting the CPython interpreter without terminating the process (e.g. embedding Python in an application) [12]_. The old ``PyThread_*_key*`` functions will be marked as deprecated in the documentation, but will not generate runtime deprecation warnings. Additionally, on platforms where ``sizeof(pthread_key_t) != sizeof(int)``, ``PyThread_create_key`` will return immediately with a failure status, and the other TLS functions will all be no-ops on such platforms. Comparison of API Specification ------------------------------- ================= ============================= ============================= API Thread Local Storage (TLS) Thread Specific Storage (TSS) ================= ============================= ============================= Version Existing New Key Type ``int`` ``Py_tss_t`` (opaque type) Handle Native Key cast to ``int`` conceal into internal field Function Argument ``int`` ``Py_tss_t *`` Features - create key - create key - delete key - delete key - set value - set value - get value - get value - delete value - (set ``NULL`` instead) [8]_ - reinitialize keys (after - (unnecessary) [8]_ fork) - dynamically (de-)allocate key - check key's initialization state Default Value (``-1`` as key creation ``Py_tss_NEEDS_INIT`` failure) Requirement native threads native threads (since CPython 3.7 [9]_) Restriction No support for platforms Unable to statically allocate where native TLS key is keys when ``Py_LIMITED_API`` defined in a way that cannot is defined. be safely cast to ``int``. ================= ============================= ============================= Example ------- With the proposed changes, a TSS key is initialized like:: static Py_tss_t tss_key = Py_tss_NEEDS_INIT; if (PyThread_tss_create(&tss_key)) { /* ... handle key creation failure ... */ } The initialization state of the key can then be checked like:: assert(PyThread_tss_is_created(&tss_key)); The rest of the API is used analogously to the old API:: int the_value = 1; if (PyThread_tss_get(&tss_key) == NULL) { PyThread_tss_set(&tss_key, (void *)&the_value); assert(PyThread_tss_get(&tss_key) != NULL); } /* ... once done with the key ... */ PyThread_tss_delete(&tss_key); assert(!PyThread_tss_is_created(&tss_key)); When ``Py_LIMITED_API`` is defined, a TSS key must be dynamically allocated:: static Py_tss_t *ptr_key = PyThread_tss_alloc(); if (ptr_key == NULL) { /* ... handle key allocation failure ... */ } assert(!PyThread_tss_is_created(ptr_key)); /* ... once done with the key ... */ PyThread_tss_free(ptr_key); ptr_key = NULL; Platform Support Changes ======================== A new "Native Thread Implementation" section will be added to PEP 11 that states: * As of CPython 3.7, all platforms are required to provide a native thread implementation (such as pthreads or Windows) to implement the TSS API. Any TSS API problems that occur in an implementation without native threads will be closed as "won't fix". Motivation ========== The primary problem at issue here is the type of the keys (``int``) used for TLS values, as defined by the original PyThread TLS API. The original TLS API was added to Python by GvR back in 1997, and at the time the key used to represent a TLS value was an ``int``, and so it has been to the time of writing. This used CPython's own TLS implementation which long remained unused, largely unchanged, in Python/thread.c. Support for implementation of the API on top of native thread implementations (pthreads and Windows) was added much later, and the built-in implementation has been deemed no longer necessary and has since been removed [9]_. The problem with the choice of ``int`` to represent a TLS key, is that while it was fine for CPython's own TLS implementation, and happens to be compatible with Windows (which uses ``DWORD`` for the analogous data), it is not compatible with the POSIX standard for the pthreads API, which defines ``pthread_key_t`` as an opaque type not further defined by the standard (as with ``Py_tss_t`` described above) [14]_. This leaves it up to the underlying implementation how a ``pthread_key_t`` value is used to look up thread-specific data. This has not generally been a problem for Python's API, as it just happens that on Linux ``pthread_key_t`` is defined as an ``unsigned int``, and so is fully compatible with Python's TLS API--``pthread_key_t``'s created by ``pthread_create_key`` can be freely cast to ``int`` and back (well, not exactly, even this has some limitations as pointed out by issue #22206). However, as issue #25658 points out, there are at least some platforms (namely Cygwin, CloudABI, but likely others as well) which have otherwise modern and POSIX-compliant pthreads implementations, but are not compatible with Python's API because their ``pthread_key_t`` is defined in a way that cannot be safely cast to ``int``. In fact, the possibility of running into this problem was raised by MvL at the time pthreads TLS was added [2]_. It could be argued that PEP-11 makes specific requirements for supporting a new, not otherwise officially-support platform (such as CloudABI), and that the status of Cygwin support is currently dubious. However, this creates a very high barrier to supporting platforms that are otherwise Linux- and/or POSIX-compatible and where CPython might otherwise "just work" except for this one hurdle. CPython itself imposes this implementation barrier by way of an API that is not compatible with POSIX (and in fact makes invalid assumptions about pthreads). Rationale for Proposed Solution =============================== The use of an opaque type (``Py_tss_t``) to key TLS values allows the API to be compatible with all present (POSIX and Windows) and future (C11?) native TLS implementations supported by CPython, as it allows the definition of ``Py_tss_t`` to depend on the underlying implementation. Since the existing TLS API has been available in *the limited API* [13]_ for some platforms (e.g. Linux), CPython makes an effort to provide the new TSS API at that level likewise. Note, however, that the ``Py_tss_t`` definition becomes to be an opaque struct when ``Py_LIMITED_API`` is defined, because exposing ``NATIVE_TSS_KEY_T`` as part of the limited API would prevent us from switching native thread implementation without rebuilding extension modules. A new API must be introduced, rather than changing the function signatures of the current API, in order to maintain backwards compatibility. The new API also more clearly groups together these related functions under a single name prefix, ``PyThread_tss_``. The "tss" in the name stands for "thread-specific storage", and was influenced by the naming and design of the "tss" API that is part of the C11 threads API [15]_. However, this is in no way meant to imply compatibility with or support for the C11 threads API, or signal any future intention of supporting C11--it's just the influence for the naming and design. The inclusion of the special default value ``Py_tss_NEEDS_INIT`` is required by the fact that not all native TLS implementations define a sentinel value for uninitialized TLS keys. For example, on Windows a TLS key is represented by a ``DWORD`` (``unsigned int``) and its value must be treated as opaque [3]_. So there is no unsigned integer value that can be safely used to represent an uninitialized TLS key on Windows. Likewise, POSIX does not specify a sentinel for an uninitialized ``pthread_key_t``, instead relying on the ``pthread_once`` interface to ensure that a given TLS key is initialized only once per-process. Therefore, the ``Py_tss_t`` type contains an explicit ``._is_initialized`` that can indicate the key's initialization state independent of the underlying implementation. Changing ``PyThread_create_key`` to immediately return a failure status on systems using pthreads where ``sizeof(int) != sizeof(pthread_key_t)`` is intended as a sanity check: Currently, ``PyThread_create_key`` may report initial success on such systems, but attempts to use the returned key are likely to fail. Although in practice this failure occurs earlier in the interpreter initialization, it's better to fail immediately at the source of problem (``PyThread_create_key``) rather than sometime later when use of an invalid key is attempted. In other words, this indicates clearly that the old API is not supported on platforms where it cannot be used reliably, and that no effort will be made to add such support. Rejected Ideas ============== * Do nothing: The status quo is fine because it works on Linux, and platforms wishing to be supported by CPython should follow the requirements of PEP-11. As explained above, while this would be a fair argument if CPython were being to asked to make changes to support particular quirks or features of a specific platform, in this case it is a quirk of CPython that prevents it from being used to its full potential on otherwise POSIX-compliant platforms. The fact that the current implementation happens to work on Linux is a happy accident, and there's no guarantee that this will never change. * Affected platforms should just configure Python ``--without-threads``: this is no longer an option as the ``--without-threads`` option has been removed for Python 3.7 [16]_. * Affected platforms should use CPython's built-in TLS implementation instead of a native TLS implementation: This is a more acceptable alternative to the previous idea, and in fact there had been a patch to do just that [4]_. However, the built-in implementation being "slower and clunkier" in general than native implementations still needlessly hobbles performance on affected platforms. At least one other module (``tracemalloc``) is also broken if Python is built without a native TLS implementation. This idea also cannot be adopted because the built-in implementation has since been removed. * Keep the existing API, but work around the issue by providing a mapping from ``pthread_key_t`` values to ``int`` values. A couple attempts were made at this ([5]_, [6]_), but this injects needless complexity and overhead into performance-critical code on platforms that are not currently affected by this issue (such as Linux). Even if use of this workaround were made conditional on platform compatibility, it introduces platform-specific code to maintain, and still has the problem of the previous rejected ideas of needlessly hobbling performance on affected platforms. Implementation ============== An initial version of a patch [7]_ is available on the bug tracker for this issue. Since the migration to GitHub, its development has continued in the ``pep539-tss-api`` feature branch [10]_ in Masayuki Yamamoto's fork of the CPython repository on GitHub. A work-in-progress PR is available at [11]_. This reference implementation covers not only the new API implementation features, but also the client code updates needed to replace the existing TLS API with the new TSS API. Copyright ========= This document has been placed in the public domain. References and Footnotes ======================== .. [1] http://bugs.python.org/issue25658 .. [2] https://bugs.python.org/msg116292 .. [3] https://msdn.microsoft.com/en-us/library/windows/desktop/ ms686801(v=vs.85).aspx .. [4] http://bugs.python.org/file45548/configure-pthread_key_t.patch .. [5] http://bugs.python.org/file44269/issue25658-1.patch .. [6] http://bugs.python.org/file44303/key-constant-time.diff .. [7] http://bugs.python.org/file46379/pythread-tss-3.patch .. [8] https://bugs.python.org/msg298342 .. [9] http://bugs.python.org/issue30832 .. [10] https://github.com/python/cpython/compare/master... ma8ma:pep539-tss-api .. [11] https://github.com/python/cpython/pull/1362 .. [12] https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx .. [13] It is also called as "stable ABI" (https://www.python.org/dev/peps/pep-0384/) .. [14] http://pubs.opengroup.org/onlinepubs/009695399/ functions/pthread_key_create.html .. [15] http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#page=404 .. [16] https://bugs.python.org/issue31370

3 5

PEP 553: Built-in debug()
by Barry Warsaw 08 Sep '17

08 Sep '17

I’ve written a PEP proposing the addition of a new built-in function called debug(). Adding this to your code would invoke a debugger through the hook function sys.debughook(). Like the existing sys.displayhook() and sys.excepthook(), you can change sys.debughook() to point to the debugger of your choice. By default it invokes pdb.set_trace(). With this PEP instead of: foo() import pdb; pdb.set_trace() bar() you can write: foo() debug() bar() and you would drop into the debugger after foo() but before bar(). More rationale and details are provided in the PEP: https://www.python.org/dev/peps/pep-0553/ Unlike David, but like Larry, I have a prototype implementation: https://github.com/python/cpython/pull/3355 Cheers, -Barry P.S. This came to me in a nightmare on Sunday night, and the more I explored the idea the more it frightened me. I know exactly what I was dreaming about and the only way to make it all go away was to write this thing up.

14 40

Memory bitmaps for the Python cyclic garbage collector
by Neil Schemenauer 08 Sep '17

08 Sep '17

Python objects that participate in cyclic GC (things like lists, dicts, sets but not strings, ints and floats) have extra memory overhead. I think it is possible to mostly eliminate this overhead. Also, while the GC is running, this GC state is mutated, which destroys copy-on-write optimizations. This change would mostly fix that issue. All objects that participate in cyclic GC have the Py_TPFLAGS_HAVE_GC bit set in their type. That causes an extra chunk of memory to be allocated *before* the ob_refcnt struct member. This is the PyGC_Head struct. The whole object looks like this in memory (PyObject pointer is at arrow): union __gc_head *gc_next; union __gc_head *gc_prev; Py_ssize_t gc_refs; --> Py_ssize_t ob_refcnt struct _typeobject *ob_type; [rest of PyObject members] So, 24 bytes of overhead on a 64-bit machine. The smallest Python object that can have a pointer to another object (e.g. a single PyObject * member) is 48 bytes. Removing PyGC_Head would cut the size of these objects in half. Carl Shaprio questioned me today on why we use a double linked-list and not the memory bitmap. I think the answer is that there is no good reason. We use a double linked list only due to historical constraints that are no longer present. Long ago, Python objects could be allocated using the system malloc or other memory allocators. Since we could not control the memory location, bitmaps would be inefficient. Today, we allocate all Python objects via our own function. Python objects under a certain size are allocated using our own malloc, obmalloc, and are stored in memory blocks known "arenas". The PyGC_Head struct performs three functions. First, it allows the GC to find all Python objects that will be checked for cycles (i.e. follow the linked list). Second, it stores a single bit of information to let the GC know if it is safe to traverse the object, set with PyObject_GC_Track(). Finally, it has a scratch area to compute the effective reference count while tracing refs (gc_refs). Here is a sketch of how we can remove the PyGC_Head struct for small objects (say less than 512 bytes). Large objects or objects created by a different memory allocator will still have the PyGC_Head overhead. * Have memory arenas that contain only objects with the Py_TPFLAGS_HAVE_GC flag. Objects like ints, strings, etc will be in different arenas, not have bitmaps, not be looked at by the cyclic GC. * For those arenas, add a memory bitmap. The bitmap is a bit array that has a bit for each fixed size object in the arena. The memory used by the bitmap is a fraction of what is needed by PyGC_Head. E.g. an arena that holds up to 1024 objects of 48 bytes in size would have a bitmap of 1024 bits. * The bits will be set and cleared by PyObject_GC_Track/Untrack() * We also need an array of Py_ssize_t to take over the job of gc_refs. That could be allocated only when GC is working and it only needs to be the size of the number of true bits in the bitmap. Or, it could be allocated when the arena is allocated and be sized for the full arena. * Objects that are too large would still get the PyGC_Head struct allocated "in front" of the PyObject. Because they are big, the overhead is not so bad. * The GC process would work nearly the same as it does now. Rather than only traversing the linked list, we would also have to crawl over the GC object arenas, check blocks of memory that have the tracked bit set. There are a lot of smaller details to work out but I see no reason why the idea should not work. It should significantly reduce memory usage. Also, because the bitmap and gc_refs are contiguous in memory, locality will be improved. Łukasz Langa has mentioned that the current GC causes issues with copy-on-write memory in big applications. This change should solve that issue. To implement, I think the easiest path is to create new malloc to be used by small GC objects, e.g. gcmalloc.c. It would be similar to obmalloc but have the features needed to keep track of the bitmap. obmalloc has some quirks that makes it hard to use for this purpose. Once the idea is proven, gcmalloc could be merged or made to be a variation of obmalloc. Or, maybe just optimized and remain separate. obmalloc is complicated and highly optimized. So, adding additional functionality to it will be challenging. I believe this change would be ABI compatible.

5 5