import in threads: crashes & strange exceptions on dual core machines

[Klaas]

It seems clear that the import lock does not include fully-executing
the module contents.  To fix this, just import cookielib before the
threads are spawned.  Better yet, use your own locks around the
acquisition of the opener instance (this code seems fraughtfully
thread-unsafe--fix that and you solve other problems besides this one).

regards,
-Mike

robert wrote:
> I get python crashes and (in better cases) strange Python exceptions when (in most cases) importing and using cookielib lazy on demand in a thread.
> It is mainly with cookielib, but remember the problem also with other imports (e.g. urllib2 etc.).
> And again very often in all these cases where I get weired Python exceptions, the problem is around re-functions - usually during re.compile calls during import (see some of the exceptions below). But not only.
>
> Very strange: The errors occur almost only on machines with dual core/multi processors - and very very rarely on very fast single core machines (>3GHz).
>
> I'm using Python2.3.5 on Win with win32ui (build 210) - the cookielib taken from Python 2.5.
>
> I took care that I'm not starting off thread things or main application loop etc. during an import (which would cause a simple & explainable deadlock freeze on the import lock)
>
> With real OS-level crashes I know from user reports (packaged app), that these errors occur very likely early after app start - thus when lazy imports are likely to do real execution.
>
> I researched this bug for some time. I think I can meanwhile exclude (ref-count, mem.leak) problems in win32ui (the only complex extension lib I use) as cause for this. All statistics point towards import problems.
>
> Any ideas?
> Are there problems known with the import lock (Python 2.3.5) ?
>
> (I cannot easily change from Python 2.3 and it takes weeks to get significant feedback after random improvements)
>
> -robert
>
> PS:
>
> The basic pattern of usage is:
>
> ==================
> def f():
>             ...
>             opener = urlcookie_openers.get(user)
>             if not opener:
>                 import cookielib                        #<----1
>                 cj=cookielib.CookieJar()                #<----2
>                 build_opener = urllib2.build_opener
>                 httpCookieProcessor = urllib2.HTTPCookieProcessor(cj)
>                 if url2_proxy:
>                     opener = build_opener(url2_proxy,httpCookieProcessor)
>                 else:
>                     opener = build_opener(httpCookieProcessor)
>                 opener.addheaders   #$pycheck_no
>                 opener.addheaders= app_addheaders
>                 urlcookie_openers[user] = opener
>             ufile = opener.open(urllib2.Request(url,data,dict(headers)))
>             ...
>
>
> thread.start_new(f,())
> =========================
>
> Symptoms:
> __________
>
> sometimes ufile is None and other weired invalid states.
>
> typical Python exceptions when in better cases there is no OS-level crash:
>
> ---------
>
>  # Attributes randomly missing like:
>  #<----2
>
> "AttributeError: \'module\' object has no attribute \'CookieJar\'\\n"]
>
>
> ---------
>
> # weired invalid states during computation like:
> #<----1
>
> ...  File "cookielib.pyo", line 184, in ?\\n\', \'  File
> "sre.pyo", line 179, in compile\\n\', \'  File "sre.pyo", line 228, in _compile\\n\', \'  File
> "sre_compile.pyo", line 467, in compile\\n\', \'  File "sre_parse.pyo", line 624, in parse\\n\', \'
> File "sre_parse.pyo", line 317, in _parse_sub\\n\', \'  File "sre_parse.pyo", line 588, in
> _parse\\n\', \'  File "sre_parse.pyo", line 92, in closegroup\\n\', \'ValueError: list.remove(x): x
> not in list\\n\']
> ...
> 'windows', "(5, 1, 2600, 2, 'Service Pack 2')/NP=2")
>
>
> ---------
>
> #<----1
>
>
> File "cookielib.pyo", line 116, in ?\\n\', \'  File "sre.pyo", line 179, in compile\\n\', \'  File "sre.pyo", line 228, in _compile\\n\', \'  File "sre_compile.pyo", line 467, in compile\\n\', \'  File "sre_parse.pyo", line 624, in parse\\n\', \'  File "sre_parse.pyo", line 317, in _parse_sub\\n\', \'  File "sre_parse.pyo", line 494, in _parse\\n\', \'  File "sre_parse.pyo", line 140, in __setitem__\\n\', \'IndexError: list assignment index out of range\\n\']
>
> ('windows', "(5, 1, 2600, 2, 'Service Pack 2')/NP=2"
>
> ---------
>
> # weired errors in other threads:
>
> # after dlg.DoModal() in main thread
>
> File "wintools.pyo", line 115, in PreTranslateMessage\\n\', \'TypeError: an integer is required\\n\']
>
> ('windows', "(5, 1, 2600, 2, 'Service Pack 2')/NP=2")
>
> ---------
>
> # after win32ui.PumpWaitingMessages(wc.WM_PAINT, wc.WM_MOUSELAST) in main thread
> 
> \'TypeError: argument list must be a tuple\\n\'
> 
> 
> ...