import in threads: crashes & strange exceptions on dual core machines

[robert]

Steve Holden wrote:
> Klaas wrote:
>> robert wrote:
>>
>>> Klaas wrote:
>>>
>>>> It seems clear that the import lock does not include fully-executing
>>>> the module contents.  To fix this, just import cookielib before the
>>>
>>> What is the exact meaning of "not include fully-executing" - 
>>> regarding the examples "import cookielib" ?
>>> Do you really mean the import statement can return without having 
>>> executed the cookielib module code fully?
>>> (As said, a simple deadlock is not at all my problem)
>>
>>
>> No, I mean that the import lock seems to not be held while the module
>> contents are being executed (which would be why you are getting
>> partially-initialized module in sys.modules).  Perhaps it _is_ held,
>> but released at various points of the import process.  Someone more
>> knowledgable of python internals will have to answer the question of
>> what _should_ be occurring.
>>
> ... and who better than Tim Peters?
> 
>   http://mail.python.org/pipermail/python-list/2004-December/254497.html
> 

this describes also a dead lock condition, when (thread starting) main-loop code is executed already during import and not via a "module.start_loop()" scheme.

Here the situation is reverse: imports are done in threads. That should go well.

The problem seem to be module namespace corruption (partial execution?) and worse OS-level crashes. I'm using zip files on the sys.path.
(
Inspected also the Py2.3.5 zipimport scheme for that, but no indication of flaws so far.  The lock is done very early in PyImport_ImportModuleEx. Just if there would be multiple Interpreters (Py_NewInterpreter) - which I don't use - the "static void lock_import(void)" would possibly be weak. )


I made a isolated thread race test with dozens of such OP code like patterns (reloads enforced) executing in parallel and found no problem.
I've spread locks and main thread (global) imports now in my app code, but it takes time until I get success significant feedback. As I have no explanation to the point .. a low percentage task :-)

win32ui/win32api (the only extension libs) are still on my radar for general ref-count/memleak problems (I found a few in the past), but that leads to abyss. I thought that the pattern of the crashes (crash early after app start; strange exceptions frequently around cookielib / re.compile) points to something else in Python itself ...


-robert