[Python-bugs-list] [ python-Bugs-613233 ] test_threadedtempfile hangs

[noreply@sourceforge.net]

Bugs item #613233, was opened at 2002-09-23 14:57
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=613233&group_id=5470

Category: Python Library
Group: Python 2.3
Status: Closed
Resolution: Fixed
Priority: 5
Submitted By: Armin Rigo (arigo)
Assigned to: Tim Peters (tim_one)
Summary: test_threadedtempfile hangs

Initial Comment:
Python 2.3's test_threadedtempfile tends to hang at the 
beginning (on my Linux machine), due to a race condition 
with the global import lock of the interpreter. The 
problem does not show up if you run "python 
test_threadedtempfile.py" directly but it does if you 
run a module that imports test_threadedtempfile.

The race condition seems to be caused by the threads 
that this test launches, each of which wants to create a 
new Random() object. This in turn causes deadlocking 
attempts to import the 'time' module.

I suggest to change the test_threadedtempfile module, 
renaming _test() to test_main() and not actually 
starting the test automatically, as was done in 
test_threaded_import.

This change also removes the need for the buggy line 32:

  tempfile.gettempdir()   \
  # Do this now to avoid spurious races later

(I suspect that more generally the confusing import 
deadlocks would be worth another bug report, although 
I'm not sure what could be done about it. Maybe just 
detecting deadlocks before they occur and raising 
ImportErrors with a clear error message?)


----------------------------------------------------------------------

>Comment By: Armin Rigo (arigo)
Date: 2002-09-26 09:12

Message:
Logged In: YES 
user_id=4771

The problem I see is not actually with the import lock itself, 
but with the underlying semantics. Indeed, without any 
threading at all:

file spam.py:
    import egg
    x=5

file egg.py:
    import spam
    print spam.x

This succeeds if and only if your application imports egg 
before it imports spam. So the programmer has to be aware 
of these problems. That's why I was suggesting that the 
import lock might only confuse things more by introducing yet 
another semantical patch for threads.

Here is a generator-based proposal. Basically, think of each 
module execution as taking place in a different coroutine. The 
statement "import spam" for a new module "spam" starts a 
new coroutine, which is initially running, and whose end gives 
the hand back to its caller (thus emulating the way Python 
currently works). But reading a missing attribute from a 
module would trigger a different behavior. It would block the 
current coroutine and pass execution to the next active one. 
This would make the above example succeed whatever 
module is first imported (which I think is more user-friendly, 
because "module object has no 'x' attribute" can be quite 
confusing if the source obviously says there is one). If all 
unfinished coroutines end up being blocked anyway, then we 
can actually raise the AttributeError in one of them (the one 
that blocked first, to emulate the current Python way).

The above idea might be extended to cooperate with threads.

Do you think this is worth being mentioned in python-dev? Or 
was the problem already discussed at lengths?

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-09-25 20:33

Message:
Logged In: YES 
user_id=31435

Fixed, in

Lib/test/test_threadedtempfile.py; new revision: 1.5

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-09-25 20:00

Message:
Logged In: YES 
user_id=31435

I believe your analysis is on target, Armin.  I'll fix it.  It hangs 
under Windows too, BTW.  The problem appears unique to 
2.3, and was introduced when a rewrite of tempfile.py started 
to import random.

I agree that the import lock can be confusing, but also unsure 
what can be done about it.  Running gobs of code as a side 
effect of merely importing is dubious practice, so I'm not sure 
I want to make that easier <wink>.  In any case, a thread 
waiting for the import lock simply can't know whether the 
thread holding it will ever let go of it; a lock here seems the 
right solution to a different problem.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=613233&group_id=5470