On 2012-03-29, at 6:56 PM, Michael Foord wrote:

I've worked with .NET where you can interrupt threads and it was very useful. There is a complication though, if a thread is interrupted inside a finally block then vital resource cleanup can be interrupted. The way .NET solves this is to never raise the interrupt exception inside a finally block. Once a finally block is completed a pending thread interrupt exception will be raised.

That's the same problem we had in our coroutine framework (based on generators + greenlets). We've decided to modify the __code__ object of generators and add a simple counter of finally statements, and when it reaches 0 - call a special callback that checks if the interruption exception should be raised. Much cleaner way would be to embed this functionality in the python interpreter itself, but I'm not sure about the performance impact. - Yury

Wouldn't it be better to raise a SystemExit in a thread you want interrupted?

On Fri, 30 Mar 2012 21:14:43 +1100 Steven D'Aprano <steve@pearwood.info> wrote:

Matt Joiner wrote:

...

Wouldn't it be better to raise a SystemExit in a thread you want interrupted?

I must admit I'm rather confused about this suggestion. Do you mean the thread itself should raise SystemExit? If so, why not just exit the normal way?

If you mean the main thread somehow injects a SystemExit in another thread, how would you do that? Perhaps I'm missing something obvious, but I don't know how to do that.

http://docs.python.org/dev/c-api/init.html#PyThreadState_SetAsyncExc Regards Antoine.

On Thu, Mar 29, 2012 at 9:53 PM, Eli Bendersky <eliben@gmail.com> wrote:

On Thu, Mar 29, 2012 at 21:48, Andrew Svetlov <andrew.svetlov@gmail.com>wrote:

...

I propose to add Thread.interrupt() function.

<snip>

Could you specify some use cases where you believe this would be better than explicitly asking the thread to stop?

Eli

The only case I can come up with is wanting to attempt to force a "clean" shutdown of threads spawned by library code out of the main processes control that has not been instrumented with its own well placed checks to see if it is time to shut down. Given such a mechanism can't interrupt blocking system calls or extension module or other VM native language level code, I'm not super excited about implementing this given the limitations. It would be relatively easy to add, I just don't see a huge benefit. As for using SystemExit as the exception or exception base class, I believe existing code such so some well known event loops catches, logs and ignores SystemExit today... ;) -gps

On 03/30/2012 12:09 PM, Serhiy Storchaka wrote:

P. S. I've had a crazy idea. What if we allow to raise any exception, not only ThreadInterruptionError, in another thread?

From an OOP point of view this is insanity. But Python (like other dynamic languages) does not enforce strict OOP semantics anyway. -panzi

After thinking I agree with Michael Foord and others. Automatic threading interruption has good use cases, but it's unsafe. Uninterrable context manager (or decorator) is good idea, but without direct support from VM python thread is 'interruptable' in all places. For now there are no way to detect where we are — in resource cleanup code or regular block which can be interrupted. Resource cleanup contains not only finally blocks but also __exit__ methods, probably except blocks, user defined callbacks etc — there are no spec for that. As result, I see no way to introduce concept of 'block for resource cleanup' without backward incompatibility and breaking existing code. I mean that concept have to mark existing finalizers as uninterruptable but we don't know that code is finalizer.

From other point it maybe useful to just add .interrupt() method and .interrupted readonly property to Thread class. The purpose of that is to make standard flag/method as request for interruption and leave actual stopping to user code. That has not so big value as (working) automatic safe interruption but can be considered as way to standardize interruption procedure.

On Sat, Mar 31, 2012 at 1:33 PM, Michael Foord <fuzzyman@gmail.com> wrote:

On 30 March 2012 11:09, Serhiy Storchaka <storchaka@gmail.com> wrote:

...

29.03.12 22:48, Andrew Svetlov написав(ла):

...

I propose to add Thread.interrupt() function.

th.interrupt() will set a flag in ThreadState structure.

When interpreter switches to next thread it will check that flag. If flag is on then ThreadInterruptionError will be raised in thread context. If thread has blocked via threading locks (Lock, RLock, Condition, Semaphore etc) — exception is raised also.

At first glance this is a very attractive suggestion. But how about alternative GIL-less implementations? The interpreter can execute some threads at the same time.

Java has a similar mechanism Thread.interrupt(), but that works only if the thread has blocked via threading locks. There is a stronger Thread.stop(), but it is recognized as unsafe and is deprecated.

It is "unsafe" because it can interrupt finally blocks - so it is impossible to protect resource cleanup from thread interruptions. Java solved this problem by deprecating thread interrupting - .NET solved it by ensuring that a thread interrupt can't happen in a finally block (so it is "safe"). The .NET solution is better. :-)

...

It would be wrong to say that this is the way to *force* stopping of some thread. ThreadInterruptionError can and should be caught in some cases.

...

BTW, we can disable interruption mechanic by default and use it only if switched on by threading.enable_interruption()

And we need context manager for noninterrable critical sections (or for interrable non-critical sections?).

"Critical section" has a particular meaning, and by raising the exception at the interpreter level (and not using the OS thread killing) we can ensure critical sections can't be interrupted. If you just mean "important sections" then preventing thread interrupts in a finally block provides one mechanism for this. An "uninterruptable context manager" would be nice - but would probably need extra vm support and isn't essential.

Michael

...

P. S. I've had a crazy idea. What if we allow to raise any exception, not only ThreadInterruptionError, in another thread?

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

--

http://www.voidspace.org.uk/

May you do good and not evil May you find forgiveness for yourself and forgive others

May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Thanks, Andrew Svetlov

On 1 April 2012 04:50, Gregory P. Smith <greg@krypto.org> wrote:

On Sat, Mar 31, 2012 at 3:33 AM, Michael Foord <fuzzyman@gmail.com> wrote:

...

An "uninterruptable context manager" would be nice - but would probably need extra vm support and isn't essential.

I'm not so sure that would need much vm support for an uninterruptable context manager, at least in CPython 3.2 and 3.3:

Isn't something like this about it; assuming you were only talking about uninteruptable within the context of native Python code rather than whatever other extension modules or interpreter embedding code may be running on their own in C/C++/Java/C# thread land:

class UninterruptableContext: def __enter__(self, ...): self._orig_switchinterval = sys.getswitchinterval() sys.setswitchinterval(1000000000) # 31 years with no Python thread switching

def __exit__(self, ...): sys.setswitchinterval(self._orig_switchinterval)

the danger with that of course is that you could be saving an obsolete switch interval value. but I suspect it is rare to change that other than near process start time. you could document the caveat and suggest that the switch interval be set to its desired setting before using any of these context managers. or monkeypatch setswitchinterval out with a dummy when this library is imported so that it becomes the sole user and owner of that api. all of which are pretty evil-hacks to expunge from ones memory and pretend you didn't read.

the _other_ big caveat to the above is that if you do any blocking operations that release the GIL within such a context manager I think you just voluntarily give up your right to not be interrupted. Plus it depends on setswitchinterval() which is an API that we could easily discard in the future with different threading and GIL implementations.

brainstorming... its what python-ideas is for.

I have zero use cases for the uninterruptable context manager within Python. for tiny sections of C code, sure. Within a high level language... not so much. Please use finer grained locks. An uninterruptible context manager is essentially a context manager around the GIL.

Hello Gregory, I think you misunderstand what we mean by uninterruptable. It has nothing to do with *thread switching*, the interruption we are talking about is the proposed new feature where threads can be terminated by raising a ThreadInterrupt exception inside them. An uninterruptable context manager (which I'm not convinced is needed or easy to implement) simply means that a ThreadInterrupt won't be raised whilst code inside the context manager is executing. It *does not* mean that execution can't switch to another thread via the normal means. Michael

-gps

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On Mon, Apr 2, 2012 at 5:54 PM, Guido van Rossum <guido@python.org> wrote:

Perhaps off-topic, but the one thing that isn't easy to do is stopping a thread that's blocked (perhaps forever) in some blocking operation -- e.g. acquiring a lock that's been forgotten or a read on a malfunctioning socket (it happens!). Having to code those operations consistently with timeouts is a pain, so if there was a way to make those system calls return an error I'd really like that.

I'm not super worried about skipping finally-clauses, we can figure out a hack for that.

Python already has support for processing EITNR in threading synchronization objects. It's done to switch GIL to main thread if signal received when GIL acquired by some background thread. That mechanic can be easy extended for thread interruption case I think. Windows is also not a problem.

On 03/30/2012 07:31 PM, Miki Tebeka wrote:

I propose to add Thread.interrupt() function.

Does the new http://docs.python.org/dev/library/signal.html#signal.pthread_kill help in any way?

This is only available on Unix. You can't send a signal just to one thread in Windows. It's always sent to the whole process.