sending thread exceptions (was Re: RELEASED Python 2.3.1)

[Alex Martelli]

Dominic wrote:

> 
>>>I and Just canvassed widely at Europython, looking for people to
>>>suggest more use cases, and nobody came up with anything that stood
>>>up to examination beyond our basic use case of "debugging possibly
>>>buggy (nonterminating) code, in cases where we just can't run the
>>>possibly buggy code in the main thread and delegate a separate
>>>watchdog thread to the purpose of interrupting the main one"
> 
> One use case could be if you only want to use a limited number
> of threads for some reason.
> 
> Then you could interrupt a low priority task and reassign the
> thread to some more urgent task. Afterwards the old task could be
> resumed. To make this work you would have to make the code
> aware of those interrupts.

I don't think it works well: Python threads have no priorities,
so, even if you DID interrupt one thread that's working on a
"low-priority job" to feed it with a higher-priority one, OTHER
threads running low-priority jobs will happily keep stealing
CPU and other resources away from the allegedly "high-priority
job".  And if you're thinking of somehow "suspending" ALL the
threads currently deemed to be running "low-priority jobs", I
think the whole architecture sounds creaky and fragile.  I would
rather tweak (not anywhere as hard a job) module Queue to give
messages posted on Queue's a priority field; ensuring that not
ALL threads (in the pool that's peeling job requests off the
main "pending-jobs Queue") are simultaneously running long AND
low-priority jobs, so that one of them is going to respond soon
enough, is decently easy -- and you can add a global sempahore,
that high-priority jobs increment at their start and decrement
at their end, and low-priority jobs check periodically in their
main loops to ensure their work is suspended when any high-
priority task is running.


> While playing with the new feature I noticed that it
> takes a long time (>3 seconds) until the exception is thrown.
> In contrast to the possibility to interrupt the main thread
> with interrupt_main which seems not to be delayed.

Hmmmm... care to show exactly the code you've been trying?
I get EXACTLY opposite results, as follows...:


import time
import thread
import threadex

def saywhen():
    for x in xrange(100000):
        time.sleep(0.1)

def intemain():
    global when_sent
    when_sent = None
    time.sleep(1.0)
    when_sent = time.time()
    thread.interrupt_main()
    time.sleep(1.0)

im_delays = []
for i in range(10):
    tid = thread.start_new_thread(intemain, ())
    try:
        saywhen()
    except:
        when_received = time.time()
    im_delays.append(when_received - when_sent)

def get_interrupted():
    global when_received
    when_received = None
    try:
        saywhen()
    except:
        when_received = time.time()

it_delays = []
for i in range(10):
    tid = thread.start_new_thread(get_interrupted, ())
    time.sleep(1.0)
    when_sent = time.time()
    threadex.threadex(tid, KeyboardInterrupt)
    time.sleep(1.0)
    it_delays.append(when_received - when_sent)

main_id = thread.get_ident()
def intemain1():
    global when_sent
    when_sent = None
    time.sleep(1.0)
    when_sent = time.time()
    threadex.threadex(main_id, KeyboardInterrupt)
    time.sleep(1.0)

im1_delays = []
for i in range(10):
    tid = thread.start_new_thread(intemain1, ())
    try:
        saywhen()
    except:
        when_received = time.time()
    im1_delays.append(when_received - when_sent)


im_delays.sort()
im1_delays.sort()
it_delays.sort()
print 'IM:', im_delays
print 'IT:', it_delays
print 'IM1:', im1_delays




Module threadex is a tiny interface exposing as 'threadex'
the PyThreadState_SetAsyncExc function.  And the results on
my Linux (Mandrake 9.1) box are as follows...:


[alex at lancelot sae]$ python pai.py
IM: [2.4993209838867188, 2.4998600482940674, 2.4998999834060669,
2.4999450445175171, 2.4999510049819946, 2.4999560117721558,
2.4999659061431885, 2.499967098236084, 2.4999990463256836,
2.5000520944595337]
IT: [0.20004498958587646, 0.39922797679901123, 0.39999902248382568,
0.40000700950622559, 0.40000808238983154, 0.40002298355102539,
0.40002298355102539, 0.40002405643463135, 0.40003299713134766,
0.40004003047943115]
IM1: [0.10003900527954102, 0.39957892894744873, 0.40000700950622559,
0.40000796318054199, 0.40001499652862549, 0.40002000331878662,
0.40003204345703125, 0.40003299713134766, 0.40004301071166992,
0.40005004405975342]
[alex at lancelot sae]$

i.e., interrupt_main takes a very repeatable 2.5 seconds;
PyThreadState_SetAsyncExc typically 0.4 seconds, whether it's
going from main to secondary thread or viceversa, with occasional
"low peaks" of 0.1 or 0.2 seconds.  Of course, it's quite
possible that there may be something biased in my setup, or
it may be a platform issue.  But I'd be quite curious to
see the code you base your observation on.


Alex