Mailman 3 August 2000 - Python-Dev

Eureka! (Re: test_fork fails --with-thread)
by Charles G Waldman Aug. 20, 2000

Aug. 20, 2000

Last month there was a flurry of discussion, around http://www.python.org/pipermail/python-dev/2000-July/014208.html about problems arising when combining threading and forking. I've been reading through the python-dev archives and as far as I can tell this problem has not yet been resolved. Well, I think I understand what's going on and I have a patch that fixes the problem. Contrary to some folklore, you *can* use fork() in threaded code; you just have to be a bit careful about locks... Rather than write up a long-winded explanation myself, allow me to quote: ----------------------------------------------------------------- from "man pthread_atfork": ... recall that fork(2) duplicates the whole memory space, including mutexes in their current locking state, but only the calling thread: other threads are not running in the child process. Thus, if a mutex is locked by a thread other than the thread calling fork, that mutex will remain locked forever in the child process, possibly blocking the execu- tion of the child process. and from http://www.lambdacs.com/newsgroup/FAQ.html#Q120 Q120: Calling fork() from a thread > Can I fork from within a thread ? Absolutely. > If that is not explicitly forbidden, then what happens to the > other threads in the child process ? There ARE no other threads in the child process. Just the one that forked. If your application/library has background threads that need to exist in a forked child, then you should set up an "atfork" child handler (by calling pthread_atfork) to recreate them. And if you use mutexes, and want your application/library to be "fork safe" at all, you also need to supply an atfork handler set to pre-lock all your mutexes in the parent, then release them in the parent and child handlers. Otherwise, ANOTHER thread might have a mutex locked when one thread forks -- and because the owning thread doesn't exist in the child, the mutex could never be released. (And, worse, whatever data is protected by the mutex is in an unknown and inconsistent state.) ------------------------------------------------------------------- Below is a patch (I will also post this to SourceForge) Notes on the patch: 1) I didn't make use of pthread_atfork, because I don't know how portable it is. So, if somebody uses "fork" in a C extension there will still be trouble. 2) I'm deliberately not cleaning up the old lock before creating the new one, because the lock destructors also do error-checking. It might be better to add a PyThread_reset_lock function to all the thread_*.h files, but I'm hesitant to do this because of the amount of testing required. Patch: Index: Modules/signalmodule.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Modules/signalmodule.c,v retrieving revision 2.53 diff -c -r2.53 signalmodule.c *** Modules/signalmodule.c 2000/08/03 02:34:44 2.53 --- Modules/signalmodule.c 2000/08/19 03:37:52 *************** *** 667,672 **** --- 667,673 ---- PyOS_AfterFork(void) { #ifdef WITH_THREAD + PyEval_ReInitThreads(); main_thread = PyThread_get_thread_ident(); main_pid = getpid(); #endif Index: Parser/intrcheck.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Parser/intrcheck.c,v retrieving revision 2.39 diff -c -r2.39 intrcheck.c *** Parser/intrcheck.c 2000/07/31 15:28:04 2.39 --- Parser/intrcheck.c 2000/08/19 03:37:54 *************** *** 206,209 **** --- 206,212 ---- void PyOS_AfterFork(void) { + #ifdef WITH_THREAD + PyEval_ReInitThreads(); + #endif } Index: Python/ceval.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/ceval.c,v retrieving revision 2.191 diff -c -r2.191 ceval.c *** Python/ceval.c 2000/08/18 19:53:25 2.191 --- Python/ceval.c 2000/08/19 03:38:06 *************** *** 142,147 **** --- 142,165 ---- Py_FatalError("PyEval_ReleaseThread: wrong thread state"); PyThread_release_lock(interpreter_lock); } + + /* This function is called from PyOS_AfterFork to ensure that newly + created child processes don't hold locks referring to threads which + are not running in the child process. (This could also be done using + pthread_atfork mechanism, at least for the pthreads implementation) */ + void + PyEval_ReInitThreads(void) + { + if (!interpreter_lock) + return; + /*XXX Can't use PyThread_free_lock here because it does too + much error-checking. Doing this cleanly would require + adding a new function to each thread_*.h. Instead, just + create a new lock and waste a little bit of memory */ + interpreter_lock = PyThread_allocate_lock(); + PyThread_acquire_lock(interpreter_lock, 1); + main_thread = PyThread_get_thread_ident(); + } #endif /* Functions save_thread and restore_thread are always defined so

2 1

serious bug in new import X as Y code
by Sjoerd Mullender Aug. 20, 2000

Aug. 20, 2000

Your changes for the import X as Y feature introduced a serious bug: I can no longer run Python at all. The problem is that in line 2150 of compile.c com_addopname is called with a NULL last argument, and the firat thing com_addopname does is indirect off of that very argument. On my machine (and on many other machines) that results in a core dump. In case it helps, here is the stack trace. The crash happens when importing site.py. I have not made any changes to my site.py. > 0 com_addopname(c = 0x7fff1e20, op = 90, n = (nil)) ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":738, 0x1006cb58] 1 com_import_stmt(c = 0x7fff1e20, n = 0x101e2ad0) ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":2150, 0x10071ecc] 2 com_node(c = 0x7fff1e20, n = 0x101e2ad0) ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":2903, 0x10074764] 3 com_node(c = 0x7fff1e20, n = 0x101eaf68) ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":2855, 0x10074540] 4 com_node(c = 0x7fff1e20, n = 0x101e2908) ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":2864, 0x100745b0] 5 com_node(c = 0x7fff1e20, n = 0x1020d450) ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":2855, 0x10074540] 6 com_file_input(c = 0x7fff1e20, n = 0x101e28f0) ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":3137, 0x10075324] 7 compile_node(c = 0x7fff1e20, n = 0x101e28f0) ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":3241, 0x100759c0] 8 jcompile(n = 0x101e28f0, filename = 0x7fff2430 = "./../Lib/site.py", base = (nil)) ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":3400, 0x10076058] 9 PyNode_Compile(n = 0x101e28f0, filename = 0x7fff2430 = "./../Lib/site.py") ["/ufs/sjoerd/src/Python/dist/src/Python/compile.c":3378, 0x10075f7c] 10 parse_source_module(pathname = 0x7fff2430 = "./../Lib/site.py", fp = 0xfb563c8) ["/ufs/sjoerd/src/Python/dist/src/Python/import.c":632, 0x100151a4] 11 load_source_module(name = 0x7fff28d8 = "site", pathname = 0x7fff2430 = "./../Lib/site.py", fp = 0xfb563c8) ["/ufs/sjoerd/src/Python/dist/src/Python/import.c":722, 0x100154c8] 12 load_module(name = 0x7fff28d8 = "site", fp = 0xfb563c8, buf = 0x7fff2430 = "./../Lib/site.py", type = 1) ["/ufs/sjoerd/src/Python/dist/src/Python/import.c":1199, 0x1001629c] 13 import_submodule(mod = 0x101b8478, subname = 0x7fff28d8 = "site", fullname = 0x7fff28d8 = "site") ["/ufs/sjoerd/src/Python/dist/src/Python/import.c":1727, 0x10017dc4] 14 load_next(mod = 0x101b8478, altmod = 0x101b8478, p_name = 0x7fff2d04, buf = 0x7fff28d8 = "site", p_buflen = 0x7fff28d0) ["/ufs/sjoerd/src/Python/dist/src/Python/import.c":1583, 0x100174c0] 15 import_module_ex(name = (nil), globals = (nil), locals = (nil), fromlist = (nil)) ["/ufs/sjoerd/src/Python/dist/src/Python/import.c":1434, 0x10016d04] 16 PyImport_ImportModuleEx(name = 0x101d9450 = "site", globals = (nil), locals = (nil), fromlist = (nil)) ["/ufs/sjoerd/src/Python/dist/src/Python/import.c":1475, 0x10016fe0] 17 PyImport_ImportModule(name = 0x101d9450 = "site") ["/ufs/sjoerd/src/Python/dist/src/Python/import.c":1408, 0x10016c64] 18 initsite() ["/ufs/sjoerd/src/Python/dist/src/Python/pythonrun.c":429, 0x10053148] 19 Py_Initialize() ["/ufs/sjoerd/src/Python/dist/src/Python/pythonrun.c":166, 0x100529c8] 20 Py_Main(argc = 1, argv = 0x7fff2ec4) ["/ufs/sjoerd/src/Python/dist/src/Modules/main.c":229, 0x10013690] 21 main(argc = 1, argv = 0x7fff2ec4) ["/ufs/sjoerd/src/Python/dist/src/Modules/python.c":10, 0x10012f24] 22 __start() ["/xlv55/kudzu-apr12/work/irix/lib/libc/libc_n32_M4/csu/crt1text.s":177, 0x10012ec8] -- Sjoerd Mullender <sjoerd.mullender(a)oratrix.com>

6 17

segfault in sre on 64-bit plats
by Trent Mick Aug. 20, 2000

Aug. 20, 2000

Fredrik, The sre module currently segfaults on one of the tests suite tests on both Win64 and 64-bit linux: [trentm@nickel src]$ ./python -c "import sre; sre.match('(x)*', 50000*'x')" > srefail.out Segmentation fault (core dumped) I know that I can't expect you to debug this completely, as you don't have to hardware, but I was hoping you might be able to shed some light on the subject for me. This test on Win32 and Linux32 hits the recursion limit check of 10000 in SRE_MATCH(). However, on Linux64 the segfault occurs at a recursion depth of 7500. I don't want to just willy-nilly drop the recursion limit down to make the problem go away. Do you have any idea why the segfault may be occuring on 64-bit platforms? Mark (Favas), have you been having any problems with sre on your 64-bit plats? In the example above I turned VERBOSE on in _sre.c. WOuld the trace help you? Here is the last of it (the whole thing is 2MB so I am not sending it all): copy 0:1 to 15026 (2) |0x600000000020b90c|0x6000000000200d72|ENTER 7517 |0x600000000020b90e|0x6000000000200d72|MARK 0 |0x600000000020b912|0x6000000000200d72|LITERAL 120 |0x600000000020b916|0x6000000000200d73|MARK 1 |0x600000000020b91a|0x6000000000200d73|MAX_UNTIL 7515 copy 0:1 to 15028 (2) |0x600000000020b90c|0x6000000000200d73|ENTER 7518 |0x600000000020b90e|0x6000000000200d73|MARK 0 |0x600000000020b912|0x6000000000200d73|LITERAL 120 |0x600000000020b916|0x6000000000200d74|MARK 1 |0x600000000020b91a|0x6000000000200d74|MAX_UNTIL 7516 copy 0:1 to 15030 (2) |0x600000000020b90c|0x6000000000200d74|ENTER 7519 |0x600000000020b90e|0x6000000000200d74|MARK 0 |0x600000000020b912|0x6000000000200d74|LITERAL 120 |0x600000000020b916|0x6000000000200d75|MARK 1 |0x600000000020b91a|0x6000000000200d75|MAX_UNTIL 7517 copy 0:1 to 15032 (2) |0x600000000020b90c|0x6000000000200d75|ENTER 7520 |0x600000000020b90e|0x6000000000200d75|MARK 0 |0x600000000020b912|0x6000000000200d75|LITERAL 120 |0x600000000020b916|0x6000000000200d76|MARK 1 |0x600000000020b91a|0x6000000000200d76|MAX_UNTIL 7518 copy 0:1 to 15034 (2) |0x600000000020b90c|0x6000000000200d76|ENTER 7521 |0x600000000020b90e|0x6000000000200d76|MARK 0 |0x600000000020b912|0x6000000000200d76|LITERAL 120 |0x600000000020b916|0x6000000000200d77|MARK 1 |0x600000000020b91a|0x6000000000200d77|MAX_UNTIL 7519 copy 0:1 to 15036 (2) |0x600000000020b90c|0x600 Thanks, Trent -- Trent Mick TrentM(a)ActiveState.com

2 3

BSDDB 3 module now somewhat functional
by A.M. Kuchling Aug. 20, 2000

Aug. 20, 2000

The handwritten BSDDB3 module has just started actually functioning. It now runs the dbtest.py script without core dumps or reported errors. Code is at ftp://starship.python.net/pub/crew/amk/new/ ; grab db.py and the most recent _bsddb.c. I started from Greg Smith's 3.1.x port of Robin Dunn's module. You'll have to struggle a bit with integrating it into Greg's package and compiling it (replacing db.py with my version, and modifying Setup to compile _bsddb.c). I haven't integrated it more, because I'm not sure how we want to proceed with it. Robin/Greg, do you want to continue to maintain the package? ...in which I'll contribute the code to one or both of you. Or, I can take over maintaining the package, or we can try to get the module into Python 2.0, but with the feature freeze well advanced, I'm doubtful that it'll get in. Still missing: Cursor objects still aren't implemented -- Martin, if you haven't started yet, let me know and I'll charge ahead with them tomorrow. Docstrings. More careful type-checking of function objects. Finally, general tidying, re-indenting, and a careful reading to catch any stupid errors that I made. --amk

6 7

Who can make test_fork1 fail?
by Tim Peters Aug. 20, 2000

Aug. 20, 2000

Note that a patch has been posted to SourceForge that purports to solve *some* thread vs fork problems: http://sourceforge.net/patch/?func=detailpatch&patch_id=101226&group_id=5470 Since nobody has made real progress on figuring out why test_fork1 fails on some systems, would somebody who is able to make it fail please just try this patch & see what happens? understanding-is-better-than-a-fix-but-i'll-settle-for-the-latter-ly y'rs - tim

3 4

New anal crusade
by Tim Peters Aug. 19, 2000

Aug. 19, 2000

Has anyone tried compiling Python under gcc with -Wmissing-prototypes -Wstrict-prototypes ? Someone on Python-Help just complained about warnings under that mode, but it's unclear to me which version of Python they were talking about.

2 3

[Fwd: Who can make test_fork1 fail?]
by Mark Favas Aug. 19, 2000

Aug. 19, 2000

1 0

Intent to document: Cookie.py
by Moshe Zadka Aug. 19, 2000

Aug. 19, 2000

This is just a notice that I'm currently in the middle of documenting Cookie. I should be finished sometime today. This is just to stop anyone else from wasting his time -- if you got time to kill, you can write a test suite <wink> -- Moshe Zadka <moshez(a)math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez

2 1

'import as'
by Thomas Wouters Aug. 19, 2000

Aug. 19, 2000

I have two remaining issues regarding the 'import as' statement, which I'm just about ready to commit. The first one is documentation: I have documentation patches, to the ref and the libdis sections, but I can't really test them :P I *think* they are fine, though, and they aren't really complicated. Should I upload a patch for them, so Fred or someone else can look at them, or just check them in ? The other issue is the change in semantics for 'from-import'. Currently, 'IMPORT_FROM' is a single operation that retrieves a name (possible '*') from the module object at TOS, and stores it directly in the local namespace. This is contrary to 'import <module>', which pushes it onto the stack and uses a normal STORE operation to store it. It's also necessary for 'from ... import *', which can load any number of objects. After the patch, 'IMPORT_FROM' is only used to load normal names, and a new opcode, 'IMPORT_STAR' (with no argument) is used for 'from module import *'. 'IMPORT_FROM' pushes the result on the stack, instead of modifying the local namespace directly, so that it's possible to store it to a different name. This also means that a 'global' statement now has effect on objects 'imported from' a module, *except* those imported by '*'. I don't think that's a big issue. 'global' is not that heavily used, and old code mixing 'import from' and 'global' statements on the same identifier would not have been doing what the programmer intended. However, if it *is* a big issue, I can revert to an older version of the patch, that added a new bytecode to handle 'from x import y as z', and leave the bytecode for the currently valid cases unchanged. That would mean that only the '... as z' would be effected by 'global' statements. -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

7 17

PyErr_NoMemory
by Vladimir.Marangozov＠inrialpes.fr Aug. 19, 2000

Aug. 19, 2000

The current PyErr_NoMemory() function reads: PyObject * PyErr_NoMemory(void) { /* raise the pre-allocated instance if it still exists */ if (PyExc_MemoryErrorInst) PyErr_SetObject(PyExc_MemoryError, PyExc_MemoryErrorInst); else /* this will probably fail since there's no memory and hee, hee, we have to instantiate this class */ PyErr_SetNone(PyExc_MemoryError); return NULL; } thus overriding any previous exceptions unconditionally. This is a problem when the current exception already *is* PyExc_MemoryError, notably when we have a chain (cascade) of memory errors. It is a problem because the original memory error and eventually its error message is lost. I suggest to make this code look like: PyObject * PyErr_NoMemory(void) { if (PyErr_ExceptionMatches(PyExc_MemoryError)) /* already current */ return NULL; /* raise the pre-allocated instance if it still exists */ if (PyExc_MemoryErrorInst) PyErr_SetObject(PyExc_MemoryError, PyExc_MemoryErrorInst); ... If nobody sees a problem with this, I'm very tempted to check it in. Any objections? -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

4 6