Mailman 3 April 2012 - Python-ideas

Improve import mechanism for circular imports
by Victor Varvariuc April 10, 2012

April 10, 2012

Consider the following directory structure (Python 3): [test] main.py [tree] __init__.py # empty branch.py root.py main.py: print('main: Entered the module') print('main: from tree import root')from tree import root # this first finished importing module 'tree.root'# then creates local name 'root' which references that imported module# Why not creating local name 'root' at the same time when sys.modules['tree.root'] is created?# If import fails:# - either … [View More]delete the attribute 'root'# - or leave it - how it affects the runtime?from tree import branch print('main: Creating a root and a leaf attached to it') _root = root.Root()_branch = branch.Branch(_root) branch.py: print('tree.branch: Entered the module') import tree, sysprint("tree.branch: 'root' in dir(tree) ->", 'root' in dir(tree))print("tree.branch: 'tree.root' in sys.modules ->", 'tree.root' in sys.modules) print('tree.branch: from tree import root')#from tree import root # ImportError: cannot import name root import tree.root # workaroundroot = sys.modules['tree.root'] # though name 'root' does not exist yet in module 'tree', sys.modules['root.tree'] already does print('tree.branch: Defining class branch.Branch()') class Branch(): def __init__(self, _parent): assert isinstance(_parent, root.Root), 'Pass a `Root` instance' root.py: print('tree.root: Entered the module') print('tree.root: from tree import branch')from tree import branch print('tree.root: Defining class Root()') class Root(): def attach(self, _branch): assert isinstance(_branch, branch.Branch), 'Pass a `Branch` instance' self.branch = _branch Running it: vic@wic:~/projects/test$ python3 main.py main: Entered the modulemain: from tree import roottree.root: Entered the moduletree.root: from tree import branchtree.branch: Entered the moduletree.branch: 'root' in dir(tree) -> Falsetree.branch: 'tree.root' in sys.modules -> Truetree.branch: from tree import roottree.branch: Defining class branch.Branch()tree.root: Defining class Root()main: Creating a root and a leaf attached to it So, There are circular imports in this example code. Currently, `root = sys.modules['tree.root']` hack in branch.py works. Wouldn't it be useful to create attribute `root` in `main` at the same time `sys.modules['tree.root']` is created when doing `from tree import root` in main? This would solve more complex cases with when circular imports are involved, without applying such hacks. Thank you -- *Victor * [View Less]

6 9

Improve circular import
by Victor Varvariuc April 10, 2012

April 10, 2012

On Tue, Apr 10, 2012 at 12:55 PM, Oleg Broytman <phd(a)phdru.name> wrote: > Why does a submodule import the entire package instead of importing just root? import tree, sys print("tree.branch: 'root' in dir(tree) ->", 'root' in dir(tree)) print("tree.branch: 'tree.root' in sys.modules ->", 'tree.root' in sys.modules) Ignore this part - my fault. It should have been: import sys print("tree.branch: 'root' in dir(main) ->", 'root' in dir(sys.modules['__main__'])) print("tree.… [View More]

1 0

pdb.set_trace may not seem long
by Yuval Greenfield April 9, 2012

April 9, 2012

Proposal: pdb.st = pdb.set_trace ----------- I find myself typing this a lot: import pdb;pdb.set_trace() It's the int3 of python. When I want to debug an exact point in the code I use the above line. I hope I don't come off as spoiled, it's just that import pdb;pdb.pm() is so short that I can't help but wonder how better my life would be if I could do: import pdb;pdb.st() What do you guys think? I know aliasing isn't cool since TSBOAPOOOWTDI but practicality beats purity.... Yuval

5 6

I found to detect if an object is GCed is very hard within python.
by 罗勇刚(Yonggang Luo) April 8, 2012

April 8, 2012

For example, I write an C extension with two object Parent and Child, they referenced each other as member, so circular referenced, how to use Python unittest to detect that? 2012/4/7, Alec Taylor <alec.taylor6(a)gmail.com>: > Has been withdrawn... and implemented > > http://www.python.org/dev/peps/pep-0274/ > -- > http://mail.python.org/mailman/listinfo/python-list > -- 从我的移动设备发送此致礼罗勇刚 Yours sincerely, Yonggang Luo

2 1

Draft PEP on protecting finally clauses
by Paul Colomiets April 7, 2012

April 7, 2012

Hi, I've finally made a PEP. Any feedback is appreciated. -- Paul PEP: XXX Title: Protecting cleanup statements from interruptions Version: $Revision$ Last-Modified: $Date$ Author: Paul Colomiets <paul(a)colomiets.name> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 06-Apr-2012 Python-Version: 3.3 Abstract ======== This PEP proposes a way to protect python code from being interrupted inside finally statement or context manager. Rationale ========= Python … [View More]has two nice ways to do cleanup. One is a ``finally`` statement and the other is context manager (or ``with`` statement). Although, neither of them is protected from ``KeyboardInterrupt`` or ``generator.throw()``. For example:: lock.acquire() try: print('starting') do_someting() finally: print('finished') lock.release() If ``KeyboardInterrupt`` occurs just after ``print`` function is executed, lock will not be released. Similarly the following code using ``with`` statement is affected:: from threading import Lock class MyLock: def __init__(self): self._lock_impl = lock def __enter__(self): self._lock_impl.acquire() print("LOCKED") def __exit__(self): print("UNLOCKING") self._lock_impl.release() lock = MyLock() with lock: do_something If ``KeyboardInterrupt`` occurs near any of the ``print`` statements, lock will never be released. Coroutine Use Case ------------------ Similar case occurs with coroutines. Usually coroutine libraries want to interrupt coroutine with a timeout. There is a ``generator.throw()`` method for this use case, but there are no method to know is it currently yielded from inside a ``finally``. Example that uses yield-based coroutines follows. Code looks similar using any of the popular coroutine libraries Monocle [1]_, Bluelet [2]_, or Twisted [3]_. :: def run_locked() yield connection.sendall('LOCK') try: yield do_something() yield do_something_else() finally: yield connection.sendall('UNLOCK') with timeout(5): yield run_locked() In the example above ``yield something`` means pause executing current coroutine and execute coroutine ``something`` until it finished execution. So that library keeps stack of generators itself. The ``connection.sendall`` waits until socket is writable and does thing similar to what ``socket.sendall`` does. The ``with`` statement ensures that all that code is executed within 5 seconds timeout. It does so by registering a callback in main loop, which calls ``generator.throw()`` to the top-most frame in the coroutine stack when timeout happens. The ``greenlets`` extension works in similar way, except it doesn't need ``yield`` to enter new stack frame. Otherwise considerations are similar. Specification ============= Frame Flag 'f_in_cleanup' ------------------------- A new flag on frame object is proposed. It is set to ``True`` if this frame is currently in the ``finally`` suite. Internally it must be implemented as a counter of nested finally statements currently executed. The internal counter is also incremented when entering ``WITH_SETUP`` bytecode and ``WITH_CLEANUP`` bytecode, and is decremented when leaving that bytecode. This allows to protect ``__enter__`` and ``__exit__`` methods too. Function 'sys.setcleanuphook' ----------------------------- A new function for the ``sys`` module is proposed. This function sets a callback which is executed every time ``f_in_cleanup`` becomes ``False``. Callbacks gets ``frame`` as it's sole argument so it can get some evindence where it is called from. The setting is thread local and is stored inside ``PyThreadState`` structure. Inspect Module Enhancements --------------------------- Two new functions are proposed for ``inspect`` module: ``isframeincleanup`` and ``getcleanupframe``. ``isframeincleanup`` given ``frame`` object or ``generator`` object as sole argument returns the value of ``f_in_cleanup`` attribute of a frame itself or of the ``gi_frame`` attribute of a generator. ``getcleanupframe`` given ``frame`` object as sole argument returns the innermost frame which has true value of ``f_in_cleanup`` or ``None`` if no frames in the stack has the attribute set. It starts to inspect from specified frame and walks to outer frames using ``f_back`` pointers, just like ``getouterframes`` does. Example ======= Example implementation of ``SIGINT`` handler that interrupts safely might look like:: import inspect, sys, functools def sigint_handler(sig, frame) if inspect.getcleanupframe(frame) is None: raise KeyboardInterrupt() sys.setcleanuphook(functools.partial(sigint_handler, 0)) Coroutine example is out of scope of this document, because it's implemention depends very much on a trampoline (or main loop) used by coroutine library. Unresolved Issues ================= Interruption Inside With Statement Expression --------------------------------------------- Given the statement:: with open(filename): do_something() Python can be interrupted after ``open`` is called, but before ``SETUP_WITH`` bytecode is executed. There are two possible decisions: * Protect expression inside ``with`` statement. This would need another bytecode, since currently there is no delimiter at the start of ``with`` expression * Let user write a wrapper if he considers it's important for his use-case. Safe wrapper code might look like the following:: class FileWrapper(object): def __init__(self, filename, mode): self.filename = filename self.mode = mode def __enter__(self): self.file = open(self.filename, self.mode) def __exit__(self): self.file.close() Alternatively it can be written using context manager:: @contextmanager def open_wrapper(filename, mode): file = open(filename, mode) try: yield file finally: file.close() This code is safe, as first part of generator (before yield) is executed inside ``WITH_SETUP`` bytecode of caller Exception Propagation --------------------- Sometimes ``finally`` block or ``__enter__/__exit__`` method can be exited with an exception. Usually it's not a problem, since more important exception like ``KeyboardInterrupt`` or ``SystemExit`` should be thrown instead. But it may be nice to be able to keep original exception inside a ``__context__`` attibute. So cleanup hook signature may grow an exception argument:: def sigint_handler(sig, frame) if inspect.getcleanupframe(frame) is None: raise KeyboardInterrupt() sys.setcleanuphook(retry_sigint) def retry_sigint(frame, exception=None): if inspect.getcleanupframe(frame) is None: raise KeyboardInterrupt() from exception .. note:: No need to have three arguments like in ``__exit__`` method since we have a ``__traceback__`` attribute in exception in Python 3.x Although, this will set ``__cause__`` for the exception, which is not exactly what's intended. So some hidden interpeter logic may be used to put ``__context__`` attribute on every exception raised in cleanup hook. Interruption Between Acquiring Resource and Try Block ----------------------------------------------------- Example from the first section is not totally safe. Let's look closer:: lock.acquire() try: do_something() finally: lock.release() There is no way it can be fixed without modifying the code. The actual fix of this code depends very much on use case. Usually code can be fixed using a ``with`` statement:: with lock: do_something() Although, for coroutines you usually can't use ``with`` statement because you need to ``yield`` for both aquire and release operations. So code might be rewritten as following:: try: yield lock.acquire() do_something() finally: yield lock.release() The actual lock code might need more code to support this use case, but implementation is usually trivial, like check if lock has been acquired and unlock if it is. Setting Interruption Context Inside Finally Itself -------------------------------------------------- Some coroutine libraries may need to set a timeout for the finally clause itself. For example:: try: do_something() finally: with timeout(0.5): try: yield do_slow_cleanup() finally: yield do_fast_cleanup() With current semantics timeout will either protect the whole ``with`` block or nothing at all, depending on the implementation of a library. What the author is intended is to treat ``do_slow_cleanup`` as an ordinary code, and ``do_fast_cleanup`` as a cleanup (non-interruptible one). Similar case might occur when using greenlets or tasklets. This case can be fixed by exposing ``f_in_cleanup`` as a counter, and by calling cleanup hook on each decrement. Corouting library may then remember the value at timeout start, and compare it on each hook execution. But in practice example is considered to be too obscure to take in account. Alternative Python Implementations Support ========================================== We consider ``f_in_cleanup`` and implementation detail. The actual implementation may have some fake frame-like object passed to signal handler, cleanup hook and returned from ``getcleanupframe``. The only requirement is that ``inspect`` module functions work as expected on that objects. For this reason we also allow to pass a ``generator`` object to a ``isframeincleanup`` function, this disables need to use ``gi_frame`` attribute. It may need to be specified that ``getcleanupframe`` must return the same object that will be passed to cleanup hook at next invocation. Alternative Names ================= Original proposal had ``f_in_finally`` flag. The original intention was to protect ``finally`` clauses. But as it grew up to protecting ``__enter__`` and ``__exit__`` methods too, the ``f_in_cleanup`` method seems better. Although ``__enter__`` method is not a cleanup routine, it at least relates to cleanup done by context managers. ``setcleanuphook``, ``isframeincleanup`` and ``getcleanupframe`` can be unobscured to ``set_cleanup_hook``, ``is_frame_in_cleanup`` and ``get_cleanup_frame``, althought they follow convention of their respective modules. Alternative Proposals ===================== Propagating 'f_in_cleanup' Flag Automatically ----------------------------------------------- This can make ``getcleanupframe`` unnecessary. But for yield based coroutines you need to propagate it yourself. Making it writable leads to somewhat unpredictable behavior of ``setcleanuphook`` Add Bytecodes 'INCR_CLEANUP', 'DECR_CLEANUP' -------------------------------------------- These bytecodes can be used to protect expression inside ``with`` statement, as well as making counter increments more explicit and easy to debug (visible inside a disassembly). Some middle ground might be chosen, like ``END_FINALLY`` and ``SETUP_WITH`` imlicitly decrements counter (``END_FINALLY`` is present at end of ``with`` suite). Although, adding new bytecodes must be considered very carefully. Expose 'f_in_cleanup' as a Counter ---------------------------------- The original intention was to expose minimum needed functionality. Although, as we consider frame flag ``f_in_cleanup`` as an implementation detail, we may expose it as a counter. Similarly, if we have a counter we may need to have cleanup hook called on every counter decrement. It's unlikely have much performance impact as nested finally clauses are unlikely common case. Add code object flag 'CO_CLEANUP' --------------------------------- As an alternative to set flag inside ``WITH_SETUP``, and ``WITH_CLEANUP`` bytecodes we can introduce a flag ``CO_CLEANUP``. When interpreter starts to execute code with ``CO_CLEANUP`` set, it sets ``f_in_cleanup`` for the whole function body. This flag is set for code object of ``__enter__`` and ``__exit__`` special methods. Technically it might be set on functions called ``__enter__`` and ``__exit__``. This seems to be less clear solution. It also covers the case where ``__enter__`` and ``__exit__`` are called manually. This may be accepted either as feature or as a unnecessary side-effect (unlikely as a bug). It may also impose a problem when ``__enter__`` or ``__exit__`` function are implemented in C, as there usually no frame to check for ``f_in_cleanup`` flag. Have Cleanup Callback on Frame Object Itself ---------------------------------------------- Frame may be extended to have ``f_cleanup_callback`` which is called when ``f_in_cleanup`` is reset to 0. It would help to register different callbacks to different coroutines. Despite apparent beauty. This solution doesn't add anything. As there are two primary use cases: * Set callback in signal handler. The callback is inherently single one for this case * Use single callback per loop for coroutine use case. And in almost all cases there is only one loop per thread No Cleanup Hook --------------- Original proposal included no cleanup hook specification. As there are few ways to achieve the same using current tools: * Use ``sys.settrace`` and ``f_trace`` callback. It may impose some problem to debugging, and has big performance impact (although, interrupting doesn't happen very often) * Sleep a bit more and try again. For coroutine library it's easy. For signals it may be achieved using ``alert``. Both methods are considered too impractical and a way to catch exit from ``finally`` statement is proposed. References ========== .. [1] Monocle https://github.com/saucelabs/monocle .. [2] Bluelet https://github.com/sampsyo/bluelet .. [3] Twisted: inlineCallbacks http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.html .. [4] Original discussion http://mail.python.org/pipermail/python-ideas/2012-April/014705.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

3 3

Protecting finally clauses of interruptions
by Paul Colomiets April 7, 2012

April 7, 2012

Hi, I'd like to propose a way to protect `finally` clauses from interruptions (either by KeyboardInterrupt or by timeout, or any other way). I think frame may be extended to have `f_in_finally` attribute (or pick a better name). Internally it should probably be implemented as a counter of nested finally clauses, but interface should probably expose only boolean attribute. For `__exit__` method some flag in `co_flags` should be introduced, which says that for whole function `f_in_finally` … [View More]

3 7

comparison of operator.itemgetter objects
by Max Moroz April 5, 2012

April 5, 2012

Currently, __eq__() method is not defined in class operator.itemgetter, hence non-identical itemgetter objects compare as non-equal. I wanted to propose defining __eq__() method that would return the result of comparison for equality of the list of arguments submitted at initialization. This would make operator.itemgetter('name') compare as equal to operator.itemgetter('name'). The motivation for this is that sorted data structure (such as blist.sortedset) might want to verify if two … [View More]arguments (say, lhs and rhs) of a binary operation (such as union) have the same sort key (a callable object passed to the constructor of the sorted data structure). Such a verification is useful because the desirable behavior of such binary operations is to use the common sort key if the lhs and rhs have the same sort key; and to raise an exception (or at least use a default value of the sort key) otherwise. I think that comparing sort keys for equality works well in many useful cases: (a) Named function. These compare as equal only if they are identical. If lhs and rhs were initialized with distinct named functions, I would argue that the programmer did not intend them to be compatible for the purpose of binary operations, even if they happen to be identical in behavior (e.g., if both functions return back the argument passed to them). In a well-designed program, there is no need to duplicate the named function definition if the two are expected to always have the same behavior. Therefore, the two distinct functions are intended to be different in behavior at least in some situations, and therefore the sorted data structure objects that use them as keys should be considered incompatible. (b) User-defined callable class. The author of such class should define __eq__() in a way that would compare as equal callable objects that behave identically, assuming it's not prohibitively expensive. Unfortunately, in two cases comparing keys for equality does not work well. (c) itemgetter. Suppose a programmer passed `itemgetter('name')` as the sort key argument to the sorted data structure's constructor. The resulting data structures would seem incompatible for the purposes of binary operations. This is likely to be confusing and undesirable. (d) lambda functions. Similarly, suppose a programmer passed `lambda x : -x` as the sort key argument to the sorted data structure's constructor. Since two lambda functions are not identical, they would compare as unequal. It seems to be very easy to address the undesirable behavior described in (c): add method __eq__() to operator.itemgetter, which would compare the list of arguments received at initialization. This would only break code that relies on an undocumented fact that distinct itemgetter instances compare as non-equal. The alternative is for each sorted data structure to handle this comparison on its own. This is repetitive and error-prone. Furthermore, it is expensive for an outsider to find out what arguments were given to an itemgetter at initialization. It is far harder to address the undesirable behavior described in (d). If it can be addressed at all, it would have to done in the sorted data structure implementation, since I don't think anyone would want lambda function comparison behavior to change. So for the purposes of this discussion, I ignore case (d). Is this a reasonable idea? Is it useful enough to be considered? Are there any downsides I didn't think of? Are there any other callables created by Python's builtin or standard library functions where __eq__ might be useful to define? Thanks, Max [View Less]

5 5

Re: [Python-ideas] Protecting finally clauses of interruptions
by Paul Colomiets April 4, 2012

April 4, 2012

Hi Yury, On Tue, Apr 3, 2012 at 1:20 AM, Yury Selivanov <yselivanov.ml(a)gmail.com> wrote: > On 2012-04-02, at 6:00 PM, Paul Colomiets wrote: > >> Hi Yury, >> >>> On 2012-04-02, at 4:49 PM, Paul Colomiets wrote: >>>> l.lock() >>>> try: >>>> ... >>>> finally: >>>> l.unlock() >>>> >>>> Which will break if you interrupted just after lock is acquired. >>> >… [View More]

5 24

Python probe: execute code in isolation (subinterpreter?) and get results
by anatoly techtonik April 4, 2012

April 4, 2012

Hi, Is there a standard way to execute a Python code and inspect the results without spawning an external Python process? If there is no such way, I'd like to propose the feature, and there are two user stories. Both are about application environment probing. Story #1: Choosing the best library in a safe manner Probing environment is required for Python applications to make component selection logic explicit and less error-prone. I can tell from my experience with Spyder IDE that startup … [View More]

3 2

dict.items to accept optional iterable with keys to use
by Victor Varvariuc April 4, 2012

April 4, 2012

Sometimes you want a dict which is subset of another dict. It would nice if dict.items accepted an optional list of keys to return. If no keys are given - use default behavior - get all items. class NewDict(dict): def items(self, keys=()): """Another version of dict.items() which accepts specific keys to use.""" for key in keys or self.keys(): yield key, self[key] a = NewDict({ 1: 'one', 2: 'two', 3: 'three', 4: 'four', 5: 'five'}) … [View More]

4 3

with *context_managers:
by Ram Rachum April 3, 2012

April 3, 2012

I'd like to be able to do this: with *context_managers: pass # Some suite. This is useful when you have an unknown number of context managers that you want to use. I currently use `contextlib.nested`, but I'd like the *star syntax much better. What do you think? Ram.

8 7

Thread stopping
by Andrew Svetlov April 2, 2012

April 2, 2012

I propose to add Thread.interrupt() function. th.interrupt() will set a flag in ThreadState structure. When interpreter switches to next thread it will check that flag. If flag is on then ThreadInterruptionError will be raised in thread context. If thread has blocked via threading locks (Lock, RLock, Condition, Semaphore etc) — exception is raised also. Of course we cannot interrupt thread if it has been locked by C Extension call or just waiting for blocking IO. But, I think, the way to … [View More]

14 22

Re: [Python-ideas] Python-ideas Digest, Vol 65, Issue 3
by Max Moroz April 2, 2012

April 2, 2012

Steven D'Aprano wrote: > In this case I was going to say just write your own subclass, but: > > py> from operator import itemgetter > py> class MyItemgetter(itemgetter): > ... pass > ... > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: type 'operator.itemgetter' is not an acceptable base type I suspect it's the same reason that bool or generator can't be subclassed: there is no obvious use … [View More]

1 0