On 29 March 2012 20:48, Andrew Svetlov wrote:

I propose to add Thread.interrupt() function.

th.interrupt() will set a flag in ThreadState structure.

When interpreter switches to next thread it will check that flag. If flag is on then ThreadInterruptionError will be raised in thread context. If thread has blocked via threading locks (Lock, RLock, Condition, Semaphore etc) — exception is raised also.

I've worked with .NET where you can interrupt threads and it was very useful. There is a complication though, if a thread is interrupted inside a finally block then vital resource cleanup can be interrupted. The way .NET solves this is to never raise the interrupt exception inside a finally block. Once a finally block is completed a pending thread interrupt exception will be raised. The normal response to requests like this is for people to suggest that the thread itself should check if it has been requested to stop - this is fine for fine grained tasks but not for very coarse grained tasks. Michael Foord

Of course we cannot interrupt thread if it has been locked by C Extension call or just waiting for blocking IO. But, I think, the way to force stopping of some thread can be useful and has no incompatibility effect.

The standard way to stop thread is the sending some message which is the signal to thread for termination. Pushing None or sentinel into thread message queue for example.

Other variants: — check 'interrupted' state explicitly by call threading.current_thread().interrupted() than do what you want. — do the same as boost.threading does: check state in direct interruption point and locks if interruption is enabled.

BTW, we can disable interruption mechanic by default and use it only if switched on by threading.enable_interruption()

What do you think?

-- Thanks, Andrew Svetlov _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Hi Yury, On Fri, Mar 30, 2012 at 2:10 AM, Yury Selivanov wrote:

Much cleaner way would be to embed this functionality in the python interpreter itself, but I'm not sure about the performance impact.

Do you have exact semantics in mind? Can you start a concrete proposal? We have same problem, and don't want to patch code object :) But I'm not sure I have clear idea of the implementation. Also I want something that works for greenlets too. -- Paul

On Thu, Mar 29, 2012 at 21:48, Andrew Svetlov wrote:

I propose to add Thread.interrupt() function.

<snip> Could you specify some use cases where you believe this would be better than explicitly asking the thread to stop? Eli

Matt Joiner wrote:

Wouldn't it be better to raise a SystemExit in a thread you want interrupted?

I must admit I'm rather confused about this suggestion. Do you mean the thread itself should raise SystemExit? If so, why not just exit the normal way? If you mean the main thread somehow injects a SystemExit in another thread, how would you do that? Perhaps I'm missing something obvious, but I don't know how to do that. -- Steven

I propose to add Thread.interrupt() function.

Does the new http://docs.python.org/dev/library/signal.html#signal.pthread_kill help in any way?

On 03/30/2012 07:31 PM, Miki Tebeka wrote:

I propose to add Thread.interrupt() function.

Does the new http://docs.python.org/dev/library/signal.html#signal.pthread_kill help in any way?

This is only available on Unix. You can't send a signal just to one thread in Windows. It's always sent to the whole process.

On Fri, 30 Mar 2012 21:14:43 +1100 Steven D'Aprano wrote:

Matt Joiner wrote:

...

Wouldn't it be better to raise a SystemExit in a thread you want interrupted?

I must admit I'm rather confused about this suggestion. Do you mean the thread itself should raise SystemExit? If so, why not just exit the normal way?

If you mean the main thread somehow injects a SystemExit in another thread, how would you do that? Perhaps I'm missing something obvious, but I don't know how to do that.

http://docs.python.org/dev/c-api/init.html#PyThreadState_SetAsyncExc Regards Antoine.

After thinking I agree with Michael Foord and others. Automatic threading interruption has good use cases, but it's unsafe. Uninterrable context manager (or decorator) is good idea, but without direct support from VM python thread is 'interruptable' in all places. For now there are no way to detect where we are — in resource cleanup code or regular block which can be interrupted. Resource cleanup contains not only finally blocks but also __exit__ methods, probably except blocks, user defined callbacks etc — there are no spec for that. As result, I see no way to introduce concept of 'block for resource cleanup' without backward incompatibility and breaking existing code. I mean that concept have to mark existing finalizers as uninterruptable but we don't know that code is finalizer.

From other point it maybe useful to just add .interrupt() method and .interrupted readonly property to Thread class. The purpose of that is to make standard flag/method as request for interruption and leave actual stopping to user code. That has not so big value as (working) automatic safe interruption but can be considered as way to standardize interruption procedure.

On Sat, Mar 31, 2012 at 1:33 PM, Michael Foord wrote:

On 30 March 2012 11:09, Serhiy Storchaka wrote:

...

29.03.12 22:48, Andrew Svetlov написав(ла):

...

I propose to add Thread.interrupt() function.

th.interrupt() will set a flag in ThreadState structure.

When interpreter switches to next thread it will check that flag. If flag is on then ThreadInterruptionError will be raised in thread context. If thread has blocked via threading locks (Lock, RLock, Condition, Semaphore etc) — exception is raised also.

At first glance this is a very attractive suggestion. But how about alternative GIL-less implementations? The interpreter can execute some threads at the same time.

Java has a similar mechanism Thread.interrupt(), but that works only if the thread has blocked via threading locks. There is a stronger Thread.stop(), but it is recognized as unsafe and is deprecated.

It is "unsafe" because it can interrupt finally blocks - so it is impossible to protect resource cleanup from thread interruptions. Java solved this problem by deprecating thread interrupting - .NET solved it by ensuring that a thread interrupt can't happen in a finally block (so it is "safe"). The .NET solution is better. :-)

...

It would be wrong to say that this is the way to *force* stopping of some thread. ThreadInterruptionError can and should be caught in some cases.

...

BTW, we can disable interruption mechanic by default and use it only if switched on by threading.enable_interruption()

And we need context manager for noninterrable critical sections (or for interrable non-critical sections?).

"Critical section" has a particular meaning, and by raising the exception at the interpreter level (and not using the OS thread killing) we can ensure critical sections can't be interrupted. If you just mean "important sections" then preventing thread interrupts in a finally block provides one mechanism for this. An "uninterruptable context manager" would be nice - but would probably need extra vm support and isn't essential.

Michael

...

P. S. I've had a crazy idea. What if we allow to raise any exception, not only ThreadInterruptionError, in another thread?

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

--

http://www.voidspace.org.uk/

May you do good and not evil May you find forgiveness for yourself and forgive others

May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Thanks, Andrew Svetlov

On Sat, Mar 31, 2012 at 3:33 AM, Michael Foord wrote:

An "uninterruptable context manager" would be nice - but would probably need extra vm support and isn't essential.

I'm not so sure that would need much vm support for an uninterruptable context manager, at least in CPython 3.2 and 3.3: Isn't something like this about it; assuming you were only talking about uninteruptable within the context of native Python code rather than whatever other extension modules or interpreter embedding code may be running on their own in C/C++/Java/C# thread land: class UninterruptableContext: def __enter__(self, ...): self._orig_switchinterval = sys.getswitchinterval() sys.setswitchinterval(1000000000) # 31 years with no Python thread switching def __exit__(self, ...): sys.setswitchinterval(self._orig_switchinterval) the danger with that of course is that you could be saving an obsolete switch interval value. but I suspect it is rare to change that other than near process start time. you could document the caveat and suggest that the switch interval be set to its desired setting before using any of these context managers. or monkeypatch setswitchinterval out with a dummy when this library is imported so that it becomes the sole user and owner of that api. all of which are pretty evil-hacks to expunge from ones memory and pretend you didn't read. the _other_ big caveat to the above is that if you do any blocking operations that release the GIL within such a context manager I think you just voluntarily give up your right to not be interrupted. Plus it depends on setswitchinterval() which is an API that we could easily discard in the future with different threading and GIL implementations. brainstorming... its what python-ideas is for. I have zero use cases for the uninterruptable context manager within Python. for tiny sections of C code, sure. Within a high level language... not so much. Please use finer grained locks. An uninterruptible context manager is essentially a context manager around the GIL. -gps

On 1 April 2012 04:50, Gregory P. Smith wrote:

On Sat, Mar 31, 2012 at 3:33 AM, Michael Foord wrote:

...

An "uninterruptable context manager" would be nice - but would probably need extra vm support and isn't essential.

I'm not so sure that would need much vm support for an uninterruptable context manager, at least in CPython 3.2 and 3.3:

Isn't something like this about it; assuming you were only talking about uninteruptable within the context of native Python code rather than whatever other extension modules or interpreter embedding code may be running on their own in C/C++/Java/C# thread land:

class UninterruptableContext: def __enter__(self, ...): self._orig_switchinterval = sys.getswitchinterval() sys.setswitchinterval(1000000000) # 31 years with no Python thread switching

def __exit__(self, ...): sys.setswitchinterval(self._orig_switchinterval)

the danger with that of course is that you could be saving an obsolete switch interval value. but I suspect it is rare to change that other than near process start time. you could document the caveat and suggest that the switch interval be set to its desired setting before using any of these context managers. or monkeypatch setswitchinterval out with a dummy when this library is imported so that it becomes the sole user and owner of that api. all of which are pretty evil-hacks to expunge from ones memory and pretend you didn't read.

the _other_ big caveat to the above is that if you do any blocking operations that release the GIL within such a context manager I think you just voluntarily give up your right to not be interrupted. Plus it depends on setswitchinterval() which is an API that we could easily discard in the future with different threading and GIL implementations.

brainstorming... its what python-ideas is for.

I have zero use cases for the uninterruptable context manager within Python. for tiny sections of C code, sure. Within a high level language... not so much. Please use finer grained locks. An uninterruptible context manager is essentially a context manager around the GIL.

Hello Gregory, I think you misunderstand what we mean by uninterruptable. It has nothing to do with *thread switching*, the interruption we are talking about is the proposed new feature where threads can be terminated by raising a ThreadInterrupt exception inside them. An uninterruptable context manager (which I'm not convinced is needed or easy to implement) simply means that a ThreadInterrupt won't be raised whilst code inside the context manager is executing. It *does not* mean that execution can't switch to another thread via the normal means. Michael

-gps

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On Mon, Apr 2, 2012 at 5:54 PM, Guido van Rossum wrote:

Perhaps off-topic, but the one thing that isn't easy to do is stopping a thread that's blocked (perhaps forever) in some blocking operation -- e.g. acquiring a lock that's been forgotten or a read on a malfunctioning socket (it happens!). Having to code those operations consistently with timeouts is a pain, so if there was a way to make those system calls return an error I'd really like that.

I'm not super worried about skipping finally-clauses, we can figure out a hack for that.

Python already has support for processing EITNR in threading synchronization objects. It's done to switch GIL to main thread if signal received when GIL acquired by some background thread. That mechanic can be easy extended for thread interruption case I think. Windows is also not a problem.