Mailman 3 April 2012 - Python-ideas

Protecting finally clauses of interruptions
by Paul Colomiets April 7, 2012

April 7, 2012

Hi, I'd like to propose a way to protect `finally` clauses from interruptions (either by KeyboardInterrupt or by timeout, or any other way). I think frame may be extended to have `f_in_finally` attribute (or pick a better name). Internally it should probably be implemented as a counter of nested finally clauses, but interface should probably expose only boolean attribute. For `__exit__` method some flag in `co_flags` should be introduced, which says that for whole function `f_in_finally` should be true. Having this attribute you can then inspect stack and check whether it's safe to interrupt it or not. Coroutine library which interrupts by timeout, can then sleep a bit and try again (probably for finite number of retries). For signal handler there are also several options to wait when thread escapes finally clause: use another thread, use alert signal, use sys.settrace, or exit only inside main loop. To be clear: I do not propose to change default SIGINT behavior, only to implement a frame flag, and give library developers experiment with the rest. -- Paul

3 7

comparison of operator.itemgetter objects
by Max Moroz April 5, 2012

April 5, 2012

Currently, __eq__() method is not defined in class operator.itemgetter, hence non-identical itemgetter objects compare as non-equal. I wanted to propose defining __eq__() method that would return the result of comparison for equality of the list of arguments submitted at initialization. This would make operator.itemgetter('name') compare as equal to operator.itemgetter('name'). The motivation for this is that sorted data structure (such as blist.sortedset) might want to verify if two arguments (say, lhs and rhs) of a binary operation (such as union) have the same sort key (a callable object passed to the constructor of the sorted data structure). Such a verification is useful because the desirable behavior of such binary operations is to use the common sort key if the lhs and rhs have the same sort key; and to raise an exception (or at least use a default value of the sort key) otherwise. I think that comparing sort keys for equality works well in many useful cases: (a) Named function. These compare as equal only if they are identical. If lhs and rhs were initialized with distinct named functions, I would argue that the programmer did not intend them to be compatible for the purpose of binary operations, even if they happen to be identical in behavior (e.g., if both functions return back the argument passed to them). In a well-designed program, there is no need to duplicate the named function definition if the two are expected to always have the same behavior. Therefore, the two distinct functions are intended to be different in behavior at least in some situations, and therefore the sorted data structure objects that use them as keys should be considered incompatible. (b) User-defined callable class. The author of such class should define __eq__() in a way that would compare as equal callable objects that behave identically, assuming it's not prohibitively expensive. Unfortunately, in two cases comparing keys for equality does not work well. (c) itemgetter. Suppose a programmer passed `itemgetter('name')` as the sort key argument to the sorted data structure's constructor. The resulting data structures would seem incompatible for the purposes of binary operations. This is likely to be confusing and undesirable. (d) lambda functions. Similarly, suppose a programmer passed `lambda x : -x` as the sort key argument to the sorted data structure's constructor. Since two lambda functions are not identical, they would compare as unequal. It seems to be very easy to address the undesirable behavior described in (c): add method __eq__() to operator.itemgetter, which would compare the list of arguments received at initialization. This would only break code that relies on an undocumented fact that distinct itemgetter instances compare as non-equal. The alternative is for each sorted data structure to handle this comparison on its own. This is repetitive and error-prone. Furthermore, it is expensive for an outsider to find out what arguments were given to an itemgetter at initialization. It is far harder to address the undesirable behavior described in (d). If it can be addressed at all, it would have to done in the sorted data structure implementation, since I don't think anyone would want lambda function comparison behavior to change. So for the purposes of this discussion, I ignore case (d). Is this a reasonable idea? Is it useful enough to be considered? Are there any downsides I didn't think of? Are there any other callables created by Python's builtin or standard library functions where __eq__ might be useful to define? Thanks, Max

5 5

Re: [Python-ideas] Protecting finally clauses of interruptions
by Paul Colomiets April 4, 2012

April 4, 2012

Hi Yury, On Tue, Apr 3, 2012 at 1:20 AM, Yury Selivanov <yselivanov.ml(a)gmail.com> wrote: > On 2012-04-02, at 6:00 PM, Paul Colomiets wrote: > >> Hi Yury, >> >>> On 2012-04-02, at 4:49 PM, Paul Colomiets wrote: >>>> l.lock() >>>> try: >>>> ... >>>> finally: >>>> l.unlock() >>>> >>>> Which will break if you interrupted just after lock is acquired. >>> >>> I guess the best way to solve this puzzle, is to track all locks that >>> the thread acquires and release them in case of forced interruption. >> >> Same with open files, and with all other kinds of contexts. I'd go >> he route of making __enter__ also uninterruptable (and make timeout >> inside a lock itself). > > > I still don't get how exactly do you propose to handle sudden thread > interruption in your own example: > > l.lock() > # (!) the thread may be interrupted at this point > try: > ... > finally: > l.unlock() > > You don't have a 'with' statement here. > By wrapping lock into a context manager. -- Paul

5 24

Python probe: execute code in isolation (subinterpreter?) and get results
by anatoly techtonik April 4, 2012

April 4, 2012

Hi, Is there a standard way to execute a Python code and inspect the results without spawning an external Python process? If there is no such way, I'd like to propose the feature, and there are two user stories. Both are about application environment probing. Story #1: Choosing the best library in a safe manner Probing environment is required for Python applications to make component selection logic explicit and less error-prone. I can tell from my experience with Spyder IDE that startup procedure is the most fragile part for this cross-platform application, which makes use of optionally installed components on user system. Implicit import nature and inability to revert import operation makes situation complicated. Below is an example. Take a note that this is not about packaging. Spyder IDE is a Qt application that optionally embeds IPython console. Qt has two bindings - PyQt4 and PySide. PyQt4 binding has two APIs - #1 and #2. If PyQt4 is used and version of installed IPython >= 0.11, the API #2 must be chosen. So, the IPython version probing should come first. A standard way to detect IPython version is to import IPython before the rest of the application, but IPython may detect PyQt4 itself and import it too for probing version. And if Spyder uses PySide we now have a conflict with Qt libraries loaded. If there was a way to execute Python script in subinterpreter to probe all installed component versions and return results, the selection logic would be much more readable and sane. Story #2: Get settings from user script Blender uses SCons to automate builds. SCons script is written in Python and it uses execfile(filename, globals, locals) to read platform specific user script with settings. Unfortunately, execfile() is a hack that doesn't treat Python scripts the same way the interpreter treats them - for example, globals access will throw exceptions - http://bugs.python.org/issue14049 More important that users won't be able to troubleshoot the exceptions, because standalone script works as expected. Executing user script code in a subprocess will most likely negatively affect performance, which is rather critical for a build tool. Pickling and unpickling objects with global state through communication pipe may also be the source of bugs. So, having a cheap way to communicate with Python subinterpreter and get a simple dict in result will make Python more snappy. -- anatoly t.

3 2

dict.items to accept optional iterable with keys to use
by Victor Varvariuc April 4, 2012

April 4, 2012

Sometimes you want a dict which is subset of another dict. It would nice if dict.items accepted an optional list of keys to return. If no keys are given - use default behavior - get all items. class NewDict(dict): def items(self, keys=()): """Another version of dict.items() which accepts specific keys to use.""" for key in keys or self.keys(): yield key, self[key] a = NewDict({ 1: 'one', 2: 'two', 3: 'three', 4: 'four', 5: 'five'}) print(dict(a.items()))print(dict(a.items((1, 3, 5)))) vic@ubuntu:~/Desktop$ python test.py {1: 'one', 2: 'two', 3: 'three', 4: 'four', 5: 'five'}{1: 'one', 3: 'three', 5: 'five'} Thanks for the attention. -- *Victor Varvariuc*

4 3

with *context_managers:
by Ram Rachum April 3, 2012

April 3, 2012

I'd like to be able to do this: with *context_managers: pass # Some suite. This is useful when you have an unknown number of context managers that you want to use. I currently use `contextlib.nested`, but I'd like the *star syntax much better. What do you think? Ram.

8 7

Thread stopping
by Andrew Svetlov April 2, 2012

April 2, 2012

I propose to add Thread.interrupt() function. th.interrupt() will set a flag in ThreadState structure. When interpreter switches to next thread it will check that flag. If flag is on then ThreadInterruptionError will be raised in thread context. If thread has blocked via threading locks (Lock, RLock, Condition, Semaphore etc) — exception is raised also. Of course we cannot interrupt thread if it has been locked by C Extension call or just waiting for blocking IO. But, I think, the way to force stopping of some thread can be useful and has no incompatibility effect. The standard way to stop thread is the sending some message which is the signal to thread for termination. Pushing None or sentinel into thread message queue for example. Other variants: — check 'interrupted' state explicitly by call threading.current_thread().interrupted() than do what you want. — do the same as boost.threading does: check state in direct interruption point and locks if interruption is enabled. BTW, we can disable interruption mechanic by default and use it only if switched on by threading.enable_interruption() What do you think? -- Thanks, Andrew Svetlov

14 22

Re: [Python-ideas] Python-ideas Digest, Vol 65, Issue 3
by Max Moroz April 2, 2012

April 2, 2012

Steven D'Aprano wrote: > In this case I was going to say just write your own subclass, but: > > py> from operator import itemgetter > py> class MyItemgetter(itemgetter): > ... pass > ... > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: type 'operator.itemgetter' is not an acceptable base type I suspect it's the same reason that bool or generator can't be subclassed: there is no obvious use case for subclassing it, and an attempt to do so is more likely to create mistakes than produce anything useful. I actually agree that itemgetter is a very specific callable class that is unlikely to be extensible in any meaningful way. Subclassing to add an __eq__() method seems to be adding what really belongs in the base class, rather than truly extending the base class. But that's just my opinion. Even if it could be done, it's not cheap. I like this recipe on SO (after a minor fix): http://stackoverflow.com/a/9970405/336527. An alternative would be to create a dummy class that defines only __getitem__ method, and use an instance of that class to collect all the values. Either approach involves creating a new object, calling the itemgetter, collecting the values into a set-like data structure, and then comparing them.

1 0