Mailman 3 April 2012 - Python-Dev

Expose dictproxy through collections rather than the types module?
by Nick Coghlan April 21, 2012

April 21, 2012

The internal dictproxy class was recently exposed as types.MappingProxyType. Since it's not very discoverable that way, would anyone object if I moved things around so it was exposed as collections.MappingProxy instead? The main benefit to doing so is to get it into the table of specialised container types at the top of the collections module docs [1]. [1] http://docs.python.org/dev/library/collections Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 3

Failed issue tracker submission
by Python tracker April 21, 2012

April 21, 2012

An unexpected error occurred during the processing of your message. The tracker administrator is being notified. Return-Path: <python-dev(a)python.org> X-Original-To: report(a)bugs.python.org Delivered-To: roundup+tracker(a)psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [82.94.164.166]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id 833861CBB0 for <report(a)bugs.python.org>; Fri, 20 Apr 2012 19:00:09 +0200 (CEST) Received: from albatross.python.org … [View More]

3 3

Re: [Python-Dev] [Python-checkins] cpython: Issue #14581: Windows users are allowed to import modules w/o taking
by Terry Reedy April 20, 2012

April 20, 2012

On 4/20/2012 12:53 PM, brett.cannon wrote: > http://hg.python.org/cpython/rev/a32be109bd86 > changeset: 76428:a32be109bd86 > user: Brett Cannon<brett(a)python.org> > date: Fri Apr 20 12:53:14 2012 -0400 > summary: > Issue #14581: Windows users are allowed to import modules w/o taking > the file suffix's case into account, even when doing a case-sensitive > import. > + name, dot, suffix = item.partition('.') Should this be .… [View More]

1 0

Summary of Python tracker Issues
by Python tracker April 20, 2012

April 20, 2012

ACTIVITY SUMMARY (2012-04-13 - 2012-04-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3396 (+19) closed 23015 (+44) total 26411 (+63) Open issues with patches: 1440 Issues opened (45) ================== #12947: Examples in library/doctest.html lack the flags http://bugs.python.org/issue12947 reopened by eric.araujo #13994: incomplete revert … [View More]

1 0

importlib is now bootstrapped (and what that means)
by Brett Cannon April 20, 2012

April 20, 2012

My multi-year project -- started in 2006 according to my blog -- to rewrite import in pure Python and then bootstrap it into CPython as *the* implementation of __import__() is finally over (mostly)! Hopefully I didn't break too much code in the process. =) Now this is "mostly" finished because the single incompatibility that importlib has is that it doesn't check the Windows registry for paths to search since I lack a Windows installation to develop and test on. If someone can tackle that … [View More]issue that would be greatly appreciated ( http://bugs.python.org/issue14578). Next up is how to maintain/develop for all of this. So the Makefile will regenerate Python/importlib.h whenever Lib/importlib/_bootstrap.py or Python/freeze_importlib.py changes. So if you make a change to importlib make sure to get it working and tested before running 'make' again else you will generate a bad frozen importlib (if you do mess up you can also revert the changes to importlib.h and re-run make; a perk to having importlib.h checked in). Otherwise keep in mind that you can't use any module that isn't a builtin (sys.builtin_module_names) in importlib._bootstrap since you can't import something w/o import working. =) Where does this leave imp and Python/import.c? I want to make imp into _imp and then implement as much as possible in pure Python (either in importlib itself or in Lib/imp.py). Once that has happened then more C code in import.c can be gutted (see http://bugs.python.org/issue13959 for tracking this work which I will start piecemeal shortly). I have some ideas on how to improve things for import, but I'm going to do them as separate emails to have separate discussion threads on them so all of this is easier to follow (e.g. actually following through on PEP 302 and exposing the import machinery as importers instead of having anything be implicit, etc.). And the only outstanding point of contention in all of this is that some people don't like having freeze_importlib.py in Python/ and instead want it in Tools/. I didn't leave it in Tools/ as I have always viewed that Python should build w/o the Tools directory, but maybe the Linux distros actually ship with it and thus this is an unneeded worry. Plus the scripts to generate the AST are in Parser so there is precedent for what I have done. Anyway, I will write up the What's New entry and double-check the language spec for updating once all of the potential changes I want to talk about in other emails have been resolved. [View Less]

10 36

issue 9141, finalizers and gc module
by Kristján Valur Jónsson April 20, 2012

April 20, 2012

Hello there. For those familiar with the intricacies of the gcmodule.c, I would like to draw your attention to http://bugs.python.org/issue9141. I would like to consult with you to find out more about finalizers/gc in order to improve the in-file documentation. Traditionally, it has not been possible to collect objects that have __del__ methods, or more generally, finalizers. Instead, they and any objects that are reachable from them, are put in gc.garbage. What I want to know is, why is … [View More]

4 8

PEP-419: Protecting cleanup statements from interruptions
by Paul Colomiets April 19, 2012

April 19, 2012

Hi, I present my first PEP. http://www.python.org/dev/peps/pep-0419/ Added text to the end of email for easier reference. Comments are welcome. -- Paul PEP: 419 Title: Protecting cleanup statements from interruptions Version: $Revision$ Last-Modified: $Date$ Author: Paul Colomiets <paul(a)colomiets.name> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 06-Apr-2012 Python-Version: 3.3 Abstract ======== This PEP proposes a way to protect Python code from … [View More]being interrupted inside a finally clause or during context manager cleanup. Rationale ========= Python has two nice ways to do cleanup. One is a ``finally`` statement and the other is a context manager (usually called using a ``with`` statement). However, neither is protected from interruption by ``KeyboardInterrupt`` or ``GeneratorExit`` caused by ``generator.throw()``. For example:: lock.acquire() try: print('starting') do_something() finally: print('finished') lock.release() If ``KeyboardInterrupt`` occurs just after the second ``print()`` call, the lock will not be released. Similarly, the following code using the ``with`` statement is affected:: from threading import Lock class MyLock: def __init__(self): self._lock_impl = Lock() def __enter__(self): self._lock_impl.acquire() print("LOCKED") def __exit__(self): print("UNLOCKING") self._lock_impl.release() lock = MyLock() with lock: do_something If ``KeyboardInterrupt`` occurs near any of the ``print()`` calls, the lock will never be released. Coroutine Use Case ------------------ A similar case occurs with coroutines. Usually coroutine libraries want to interrupt the coroutine with a timeout. The ``generator.throw()`` method works for this use case, but there is no way of knowing if the coroutine is currently suspended from inside a ``finally`` clause. An example that uses yield-based coroutines follows. The code looks similar using any of the popular coroutine libraries Monocle [1]_, Bluelet [2]_, or Twisted [3]_. :: def run_locked(): yield connection.sendall('LOCK') try: yield do_something() yield do_something_else() finally: yield connection.sendall('UNLOCK') with timeout(5): yield run_locked() In the example above, ``yield something`` means to pause executing the current coroutine and to execute coroutine ``something`` until it finishes execution. Therefore the coroutine library itself needs to maintain a stack of generators. The ``connection.sendall()`` call waits until the socket is writable and does a similar thing to what ``socket.sendall()`` does. The ``with`` statement ensures that all code is executed within 5 seconds timeout. It does so by registering a callback in the main loop, which calls ``generator.throw()`` on the top-most frame in the coroutine stack when a timeout happens. The ``greenlets`` extension works in a similar way, except that it doesn't need ``yield`` to enter a new stack frame. Otherwise considerations are similar. Specification ============= Frame Flag 'f_in_cleanup' ------------------------- A new flag on the frame object is proposed. It is set to ``True`` if this frame is currently executing a ``finally`` clause. Internally, the flag must be implemented as a counter of nested finally statements currently being executed. The internal counter also needs to be incremented during execution of the ``SETUP_WITH`` and ``WITH_CLEANUP`` bytecodes, and decremented when execution for these bytecodes is finished. This allows to also protect ``__enter__()`` and ``__exit__()`` methods. Function 'sys.setcleanuphook' ----------------------------- A new function for the ``sys`` module is proposed. This function sets a callback which is executed every time ``f_in_cleanup`` becomes false. Callbacks get a frame object as their sole argument, so that they can figure out where they are called from. The setting is thread local and must be stored in the ``PyThreadState`` structure. Inspect Module Enhancements --------------------------- Two new functions are proposed for the ``inspect`` module: ``isframeincleanup()`` and ``getcleanupframe()``. ``isframeincleanup()``, given a frame or generator object as its sole argument, returns the value of the ``f_in_cleanup`` attribute of a frame itself or of the ``gi_frame`` attribute of a generator. ``getcleanupframe()``, given a frame object as its sole argument, returns the innermost frame which has a true value of ``f_in_cleanup``, or ``None`` if no frames in the stack have a nonzero value for that attribute. It starts to inspect from the specified frame and walks to outer frames using ``f_back`` pointers, just like ``getouterframes()`` does. Example ======= An example implementation of a SIGINT handler that interrupts safely might look like:: import inspect, sys, functools def sigint_handler(sig, frame): if inspect.getcleanupframe(frame) is None: raise KeyboardInterrupt() sys.setcleanuphook(functools.partial(sigint_handler, 0)) A coroutine example is out of scope of this document, because its implementation depends very much on a trampoline (or main loop) used by coroutine library. Unresolved Issues ================= Interruption Inside With Statement Expression --------------------------------------------- Given the statement :: with open(filename): do_something() Python can be interrupted after ``open()`` is called, but before the ``SETUP_WITH`` bytecode is executed. There are two possible decisions: * Protect ``with`` expressions. This would require another bytecode, since currently there is no way of recognizing the start of the ``with`` expression. * Let the user write a wrapper if he considers it important for the use-case. A safe wrapper might look like this:: class FileWrapper(object): def __init__(self, filename, mode): self.filename = filename self.mode = mode def __enter__(self): self.file = open(self.filename, self.mode) def __exit__(self): self.file.close() Alternatively it can be written using the ``contextmanager()`` decorator:: @contextmanager def open_wrapper(filename, mode): file = open(filename, mode) try: yield file finally: file.close() This code is safe, as the first part of the generator (before yield) is executed inside the ``SETUP_WITH`` bytecode of the caller. Exception Propagation --------------------- Sometimes a ``finally`` clause or an ``__enter__()``/``__exit__()`` method can raise an exception. Usually this is not a problem, since more important exceptions like ``KeyboardInterrupt`` or ``SystemExit`` should be raised instead. But it may be nice to be able to keep the original exception inside a ``__context__`` attribute. So the cleanup hook signature may grow an exception argument:: def sigint_handler(sig, frame) if inspect.getcleanupframe(frame) is None: raise KeyboardInterrupt() sys.setcleanuphook(retry_sigint) def retry_sigint(frame, exception=None): if inspect.getcleanupframe(frame) is None: raise KeyboardInterrupt() from exception .. note:: There is no need to have three arguments like in the ``__exit__`` method since there is a ``__traceback__`` attribute in exception in Python 3. However, this will set the ``__cause__`` for the exception, which is not exactly what's intended. So some hidden interpreter logic may be used to put a ``__context__`` attribute on every exception raised in a cleanup hook. Interruption Between Acquiring Resource and Try Block ----------------------------------------------------- The example from the first section is not totally safe. Let's take a closer look:: lock.acquire() try: do_something() finally: lock.release() The problem might occur if the code is interrupted just after ``lock.acquire()`` is executed but before the ``try`` block is entered. There is no way the code can be fixed unmodified. The actual fix depends very much on the use case. Usually code can be fixed using a ``with`` statement:: with lock: do_something() However, for coroutines one usually can't use the ``with`` statement because you need to ``yield`` for both the acquire and release operations. So the code might be rewritten like this:: try: yield lock.acquire() do_something() finally: yield lock.release() The actual locking code might need more code to support this use case, but the implementation is usually trivial, like this: check if the lock has been acquired and unlock if it is. Handling EINTR Inside a Finally ------------------------------- Even if a signal handler is prepared to check the ``f_in_cleanup`` flag, ``InterruptedError`` might be raised in the cleanup handler, because the respective system call returned an ``EINTR`` error. The primary use cases are prepared to handle this: * Posix mutexes never return ``EINTR`` * Networking libraries are always prepared to handle ``EINTR`` * Coroutine libraries are usually interrupted with the ``throw()`` method, not with a signal The platform-specific function ``siginterrupt()`` might be used to remove the need to handle ``EINTR``. However, it may have hardly predictable consequences, for example ``SIGINT`` a handler is never called if the main thread is stuck inside an IO routine. A better approach would be to have the code, which is usually used in cleanup handlers, be prepared to handle ``InterruptedError`` explicitly. An example of such code might be a file-based lock implementation. Setting Interruption Context Inside Finally Itself -------------------------------------------------- Some coroutine libraries may need to set a timeout for the finally clause itself. For example:: try: do_something() finally: with timeout(0.5): try: yield do_slow_cleanup() finally: yield do_fast_cleanup() With current semantics, timeout will either protect the whole ``with`` block or nothing at all, depending on the implementation of each library. What the author intended is to treat ``do_slow_cleanup`` as ordinary code, and ``do_fast_cleanup`` as a cleanup (a non-interruptible one). A similar case might occur when using greenlets or tasklets. This case can be fixed by exposing ``f_in_cleanup`` as a counter, and by calling a cleanup hook on each decrement. A coroutine library may then remember the value at timeout start, and compare it on each hook execution. But in practice, the example is considered to be too obscure to take into account. Modifying KeyboardInterrupt --------------------------- It should be decided if the default ``SIGINT`` handler should be modified to use the described mechanism. The initial proposition is to keep old behavior, for two reasons: * Most application do not care about cleanup on exit (either they do not have external state, or they modify it in crash-safe way). * Cleanup may take too much time, not giving user a chance to interrupt an application. The latter case can be fixed by allowing an unsafe break if a ``SIGINT`` handler is called twice, but it seems not worth the complexity. Alternative Python Implementations Support ========================================== We consider ``f_in_cleanup`` an implementation detail. The actual implementation may have some fake frame-like object passed to signal handler, cleanup hook and returned from ``getcleanupframe()``. The only requirement is that the ``inspect`` module functions work as expected on these objects. For this reason, we also allow to pass a generator object to the ``isframeincleanup()`` function, which removes the need to use the ``gi_frame`` attribute. It might be necessary to specify that ``getcleanupframe()`` must return the same object that will be passed to cleanup hook at the next invocation. Alternative Names ================= The original proposal had a ``f_in_finally`` frame attribute, as the original intention was to protect ``finally`` clauses. But as it grew up to protecting ``__enter__`` and ``__exit__`` methods too, the ``f_in_cleanup`` name seems better. Although the ``__enter__`` method is not a cleanup routine, it at least relates to cleanup done by context managers. ``setcleanuphook``, ``isframeincleanup`` and ``getcleanupframe`` can be unobscured to ``set_cleanup_hook``, ``is_frame_in_cleanup`` and ``get_cleanup_frame``, although they follow the naming convention of their respective modules. Alternative Proposals ===================== Propagating 'f_in_cleanup' Flag Automatically --------------------------------------------- This can make ``getcleanupframe()`` unnecessary. But for yield-based coroutines you need to propagate it yourself. Making it writable leads to somewhat unpredictable behavior of ``setcleanuphook()``. Add Bytecodes 'INCR_CLEANUP', 'DECR_CLEANUP' -------------------------------------------- These bytecodes can be used to protect the expression inside the ``with`` statement, as well as making counter increments more explicit and easy to debug (visible inside a disassembly). Some middle ground might be chosen, like ``END_FINALLY`` and ``SETUP_WITH`` implicitly decrementing the counter (``END_FINALLY`` is present at end of every ``with`` suite). However, adding new bytecodes must be considered very carefully. Expose 'f_in_cleanup' as a Counter ---------------------------------- The original intention was to expose a minimum of needed functionality. However, as we consider the frame flag ``f_in_cleanup`` an implementation detail, we may expose it as a counter. Similarly, if we have a counter we may need to have the cleanup hook called on every counter decrement. It's unlikely to have much performance impact as nested finally clauses are an uncommon case. Add code object flag 'CO_CLEANUP' --------------------------------- As an alternative to set the flag inside the ``SETUP_WITH`` and ``WITH_CLEANUP`` bytecodes, we can introduce a flag ``CO_CLEANUP``. When the interpreter starts to execute code with ``CO_CLEANUP`` set, it sets ``f_in_cleanup`` for the whole function body. This flag is set for code objects of ``__enter__`` and ``__exit__`` special methods. Technically it might be set on functions called ``__enter__`` and ``__exit__``. This seems to be less clear solution. It also covers the case where ``__enter__`` and ``__exit__`` are called manually. This may be accepted either as a feature or as an unnecessary side-effect (or, though unlikely, as a bug). It may also impose a problem when ``__enter__`` or ``__exit__`` functions are implemented in C, as there is no code object to check for the ``f_in_cleanup`` flag. Have Cleanup Callback on Frame Object Itself -------------------------------------------- The frame object may be extended to have a ``f_cleanup_callback`` member which is called when ``f_in_cleanup`` is reset to 0. This would help to register different callbacks to different coroutines. Despite its apparent beauty, this solution doesn't add anything, as the two primary use cases are: * Setting the callback in a signal handler. The callback is inherently a single one for this case. * Use a single callback per loop for the coroutine use case. Here, in almost all cases, there is only one loop per thread. No Cleanup Hook --------------- The original proposal included no cleanup hook specification, as there are a few ways to achieve the same using current tools: * Using ``sys.settrace()`` and the ``f_trace`` callback. This may impose some problem to debugging, and has a big performance impact (although interrupting doesn't happen very often). * Sleeping a bit more and trying again. For a coroutine library this is easy. For signals it may be achieved using ``signal.alert``. Both methods are considered too impractical and a way to catch exit from ``finally`` clauses is proposed. References ========== .. [1] Monocle https://github.com/saucelabs/monocle .. [2] Bluelet https://github.com/sampsyo/bluelet .. [3] Twisted: inlineCallbacks http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.html .. [4] Original discussion http://mail.python.org/pipermail/python-ideas/2012-April/014705.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

4 5

PEP 420: Implicit Namespace Packages
by Eric V. Smith April 19, 2012

April 19, 2012

If you have any comments, please join the discussion over in import-sig. Eric.

1 0

Re: [Python-Dev] [Python-checkins] cpython: Issue #11750: The Windows API functions scattered in the _subprocess and
by "Martin v. Löwis" April 19, 2012

April 19, 2012

Am 18.04.2012 20:52, schrieb antoine.pitrou: > http://hg.python.org/cpython/rev/f3a27d11101a > changeset: 76405:f3a27d11101a > user: Antoine Pitrou <solipsis(a)pitrou.net> > date: Wed Apr 18 20:51:15 2012 +0200 > summary: > Issue #11750: The Windows API functions scattered in the _subprocess and > _multiprocessing.win32 modules now live in a single module "_winapi". > Patch by sbt. Can we use Real Names, please? Regards, Martin

5 12

Re: [Python-Dev] Suggested addition to PEP 8 for context managers
by Tshepang Lekhonkhobe April 19, 2012

April 19, 2012

was sent to Barry-only by mistake On Thu, Apr 19, 2012 at 17:20, Tshepang Lekhonkhobe <tshepang(a)gmail.com> wrote: > On Thu, Apr 19, 2012 at 17:15, Barry Warsaw <barry(a)python.org> wrote: >> If I change that phrase to "Use your own judgement" does that help? > > I would prefer "This is a matter of taste...". Much closer to original > meaning, and I think it's a more common phrase.

1 0