Mailman 3 June 2004 - Python-Dev

various test failures - test_site, test_tempfile, test_bsddb/anydbm
by Skip Montanaro 08 Jun '04

08 Jun '04

I was making some simple changes to fileobject.c yesterday (checking file() mode strings for validity) and started having test failures. After lots of head scratching and changes to my code I concluded it couldn't be my code and finally just ripped out the changes and ran with a fresh CVS checkout. Sure enough, I got the same problem. On Mac OS X I get this output (debug build): ... test_sys test_tarfile test_tcl test_tempfile *** malloc[24360]: error for object 0x680b408: Pointer being reallocated was not allocated [5939 refs] test test_tempfile failed -- Traceback (most recent call last): File "/Users/skip/tmp/python/dist/src/Lib/test/test_tempfile.py", line 155, in test_wanted_dirs os.environ[envname] = os.path.abspath(envname) File "/Users/skip/tmp/python/dist/src/Lib/os.py", line 445, in __setitem__ putenv(key, item) OSError: (0, 'Error') test_textwrap test_thread test_threaded_import test_threadedtempfile test_threading test_time make: *** [test] Bus error This stuff passes with no complaint on Linux (Mandrake 8.1-ish) but I'm having other problems there. Tests involving bsddb fail (I'm building with Berkeley DB 3.3 - is that too old now?) with something like this: test_anydbm UåSìcèíÂ make: *** [quicktest] Segmentation fault On both Mac OS X and Linux (Mandrake 8.1-ish) test_site fails with: test test_site failed -- Traceback (most recent call last): File "/home/skip/tmp/python/dist/src/Lib/test/test_site.py", line 150, in test_sitepackages self.fail("'site-packages' directory missing'") AssertionError: 'site-packages' directory missing' This seems like a problem with the test itself. It should be okay to run the tests in the build directory without a site-packages directory. I'm doing a little further investigation into the malloc problem on Mac OS X using the debug stuff available in its malloc(3) implementation. Skip

3 3

fopenRF() - defunct MacOS 9 thing?
by Skip Montanaro 08 Jun '04

08 Jun '04

I was messing around in fileobject.c today and came across this: #ifdef HAVE_FOPENRF if (*mode == '*') { FILE *fopenRF(); f->f_fp = fopenRF(name, mode+1); } else #endif Googling for fopenRF() indicated it is a way to open resource forks on Macs (all the Google hits were Python-related). Is that something that should have been deleted with the demise of MacOS 9 support? Skip

2 2

Python/import.c
by Thomas Heller 08 Jun '04

08 Jun '04

In the context of http://www.python.org/sf/845802 I'm looking at the code in Python/import.c, function load_package(). The latter part of the function is this code, which tries to import the __init__.py file, after building the module object in 'm', and initializing it's __file__ and __path__ attributes: fdp = find_module(name, "__init__", path, buf, sizeof(buf), &fp, NULL); if (fdp == NULL) { if (PyErr_ExceptionMatches(PyExc_ImportError)) { PyErr_Clear(); } else m = NULL; goto cleanup; } m = load_module(name, fp, buf, fdp->type, NULL); if (fp != NULL) fclose(fp); cleanup: Py_XDECREF(path); Py_XDECREF(file); return m; } I do not understand why the function doesn't fail when find_module returns NULL because of an PyExc_ImportError. For the above mentioned bug, when __init__.py is a directory instead of a file, find_module returns NULL. Thomas

3 3

Bug day outcome
by A.M. Kuchling 07 Jun '04

07 Jun '04

The bug day yesterday was a success, despite ending four hours early when SourceForge CVS stopped working and didn't come back. (So, are we thinking of running our own CVS/SVN/whatever server?) About eight people showed up; attendance was probably reduced because some people are at the EuroPython sprints working on other things. That's still enough people to get a significant amount of work done. The last bug summary said there were 762 open bugs. 30 bugs (= 4%) were closed during the bug day, and http://www.python.org/moin/PythonBugDayStatus lists 14 more bugs that need to be reviewed by a core developer and possibly committed. IMHO the bug day was a success, and we should have a few more of them before 2.4final. Notes for the next one: * Perhaps the next one could be on the second weekend in July (the 10th/11th). Or should we try to have it before the first alpha is released? * This one was originally supposed to be 12 hours long, but that's really too long for the organizer to keep focus :). The next one should be just eight hours. * The inability for random third-party developers to attach patches to bugs was an annoyance, not a serious problem; I attached a few files for developers who weren't able to do it themselves. Still, I hope the PSF pushes forward on the Roundup work. --amk

9 15

RE: [Python-Dev] Empty keyword
by Robert Brewer 07 Jun '04

07 Jun '04

Sylvain Hellegouarch wrote: > It miht sound stupid or useless, or maybe someone has already > mentioned > it but sometimes I wish Python could provdie an "Empty" > keyword like The > "None" one. > > It would be usefull to validdate the emptyness of any scalar > or sequence > in an uniform way instead of using things such as : > > if len(mylist) == 0: > > which I really dislike. Have you read the bit about empty lists and such when testing truth values? http://docs.python.org/lib/truth.html >>> x = [] >>> if x: ... print True ... else: ... print False ... False >>> x = [0] >>> if x: ... print True ... else: ... print False ... True Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

3 3

RE: [Python-Dev] Empty keyword
by Batista, Facundo 07 Jun '04

07 Jun '04

[Sylvain Hellegouarch] #- It would be usefull to validdate the emptyness of any scalar #- or sequence #- in an uniform way instead of using things such as : #- #- if len(mylist) == 0: Taking in consideration that I'll also like such keyword, making a new keyword for a functionality that already exists like an idiom is not very likely to happen. Anyway, if you really want to push this, I recommend you to search in the archives why others keywords have been rejected in the past, and if this new one passes all the tests, make a PEP. . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La información contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener información confidencial o propietaria, cuya divulgación es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no está autorizado a divulgar, copiar, distribuir o retener información (o parte de ella) contenida en este mensaje. Por favor notifíquenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magnético) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telefónica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electrónicos pueden ser alterados, motivo por el cual Telefónica Comunicaciones Personales S.A. no aceptará ninguna obligación cualquiera sea el resultante de este mensaje. Muchas Gracias.

2 1

RE: [Python-Dev] Bug day outcome
by Tony Meyer 06 Jun '04

06 Jun '04

[Tim Peters] > I noticed just now that > > A Python Bug Day will be held on Saturday, June 5; see > the Wiki for information. > > is a prominent announcement on the python.org home page, but > I don't look at that often. Ah, I see that now, thanks :) I did look at python.org before posting, but didn't pay much attention to the main page, and looked carefully at /dev instead :) In any case, I would have been asleep (1am to 10am here) or too sleepy to contribute my meagre skills anyway :) =Tony Meyer

1 0

valgrind report
by Neal Norwitz 06 Jun '04

06 Jun '04

I checked in several changes to support valgrind better. Mostly it was documentation changes (additions). The changes should be minimal enough to be considered for backporting to 2.3.5. Although I did not mark all the checkins as such. At the end of this mail are the remaining issues that valgrind reports. There were very few problems reported. Valgrind was run on gentoo, dual amd athlon box with 1GB. I checked in a fix to Modules/binascii.c. Someone familiar with the module should review this change. It fixed an uninitialized memory read. It would be nice to improve the comment. I fixed a memory leak in Modules/posixmodule.c when callin utime(). The code is a bit complex, so it would be good to review this change carefully as well. Neal -- These are from most important to fix to least important. I submitted a patch to fix a bunch of possible memory leaks. the patch fixes at least one memory leak not reported here. The fixes are in the same location as these leaks. http://sourceforge.net/tracker/index.php?func=detail&aid=967763&group_id=54… BSD DB memory leaks: -------------------- 4 bytes in 1 blocks are definitely lost in loss record 7 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C928265: make_key_dbt (_bsddb.c:400) by 0x3C92DE4F: DBC_set_range (_bsddb.c:2911) 8 bytes in 2 blocks are definitely lost in loss record 29 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C928265: make_key_dbt (_bsddb.c:400) by 0x3C929088: DB_get (_bsddb.c:1349) I'm not sure which test caused these leaks. I will try narrow these two down. I suspect one may be test_import.py. Import/compile related memory leaks: ------------------------------------ 128 bytes in 1 blocks are definitely lost in loss record 221 of 799 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x807CDB7: new_arena (obmalloc.c:500) by 0x807CAF2: PyObject_Malloc (obmalloc.c:699) by 0x807D5A7: PyString_FromStringAndSize (stringobject.c:73) by 0x80D6D43: r_object (marshal.c:500) by 0x80D63D5: r_object (marshal.c:621) by 0x80D5E01: r_object (marshal.c:546) by 0x80D6380: r_object (marshal.c:616) by 0x80D7683: PyMarshal_ReadLastObjectFromFile (marshal.c:665) by 0x80D0CAA: load_source_module (import.c:690) by 0x80CEA8D: load_module (import.c:1616) by 0x80CF977: import_submodule (import.c:2210) by 0x80CF4A2: load_next (import.c:2030) by 0x80D14D8: import_module_ex (import.c:1865) by 0x80D054B: PyImport_ImportModuleEx (import.c:1906) 256 bytes in 1 blocks are definitely lost in loss record 252 of 799 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x807CDB7: new_arena (obmalloc.c:500) by 0x807CAF2: PyObject_Malloc (obmalloc.c:699) by 0x80F8914: PyNode_AddChild (node.c:95) by 0x80F8BE1: PyParser_AddToken (parser.c:112) by 0x8055AF2: parsetok (parsetok.c:166) by 0x80DBFAD: Py_CompileStringFlags (pythonrun.c:1375) by 0x80A2AAC: builtin_compile (bltinmodule.c:391) by 0x81019F9: PyCFunction_Call (methodobject.c:108) by 0x80AA71D: call_function (ceval.c:3377) by 0x80AE5FF: eval_frame (ceval.c:1990) by 0x80A93AD: PyEval_EvalCodeEx (ceval.c:2543) by 0x80AA898: fast_function (ceval.c:3468) by 0x80AA3F3: call_function (ceval.c:3397) by 0x80AE5FF: eval_frame (ceval.c:1990) The thread leaks may occur at most once in a program. I'm not sure. Thread memory leaks: -------------------- 16 bytes in 1 blocks are definitely lost in loss record 63 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x80DF227: PyThread_allocate_lock (thread_pthread.h:262) by 0x80ABC43: PyEval_InitThreads (ceval.c:162) by 0x80E287F: thread_PyThread_start_new_thread (threadmodule.c:243) 64 bytes in 4 blocks are definitely lost in loss record 132 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x80E2843: thread_PyThread_start_new_thread (threadmodule.c:233) Initialization memory leaks: ---------------------------- 64 bytes in 1 blocks are definitely lost in loss record 139 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x807CE32: new_arena (obmalloc.c:452) by 0x807CAF2: PyObject_Malloc (obmalloc.c:699) by 0x80E1566: _PyObject_GC_Malloc (gcmodule.c:1183) by 0x80E166C: _PyObject_GC_NewVar (gcmodule.c:1214) by 0x8084A71: PyTuple_New (tupleobject.c:68) by 0x8087B2F: PyType_Ready (typeobject.c:3161) by 0x8087B39: PyType_Ready (typeobject.c:3147) by 0x807BE01: _Py_ReadyTypes (object.c:1802) by 0x80D9B93: Py_Initialize (pythonrun.c:167) by 0x805510A: Py_Main (main.c:376) by 0x8054DAA: main (python.c:23) I don't know if we can fix any of these. They may all be within non-Python libraries. DL memory leaks: ---------------- 17 bytes in 1 blocks are definitely lost in loss record 90 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C0060D9: _dl_map_object (in /lib/ld-2.3.2.so) by 0x3C25E0AA: (within /lib/libc-2.3.2.so) by 0x3C00B247: _dl_catch_error (in /lib/ld-2.3.2.so) by 0x3C25E6C8: _dl_open (in /lib/libc-2.3.2.so) by 0x3C06DCF7: (within /lib/libdl-2.3.2.so) by 0x3C00B247: _dl_catch_error (in /lib/ld-2.3.2.so) by 0x3C06D485: (within /lib/libdl-2.3.2.so) by 0x3C06DD4B: dlopen (in /lib/libdl-2.3.2.so) by 0x3F9DD3E2: dl_open (dlmodule.c:184) Readline memory leaks: ---------------------- 40 bytes in 2 blocks are definitely lost in loss record 116 of 663 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3CB85EF1: xmalloc (in /lib/libreadline.so.4.3) Reading password/group memory leaks: ------------------------------------ 36 bytes in 1 blocks are definitely lost in loss record 162 of 799 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C23EBC2: (within /lib/libc-2.3.2.so) by 0x3C23E4F2: __nss_database_lookup (in /lib/libc-2.3.2.so) by 0x3FA3533D: ??? by 0x3FA355B0: ??? by 0x3C23F5FD: (within /lib/libc-2.3.2.so) by 0x3C200131: setgrent (in /lib/libc-2.3.2.so) by 0x3CA32E17: grp_getgrall (grpmodule.c:123) 36 bytes in 1 blocks are definitely lost in loss record 163 of 799 at 0x3C01DA1D: malloc (vg_replace_malloc.c:109) by 0x3C23EBC2: (within /lib/libc-2.3.2.so) by 0x3C23E4F2: __nss_database_lookup (in /lib/libc-2.3.2.so) by 0x3FA362FD: ??? by 0x3FA3780C: ??? by 0x3C201597: getpwnam_r (in /lib/libc-2.3.2.so) by 0x3C20103D: getpwnam (in /lib/libc-2.3.2.so) by 0x3CB48D80: pwd_getpwnam (pwdmodule.c:130)

2 1

Allowing uploads in bug tracker
by A.M. Kuchling 06 Jun '04

06 Jun '04

Is there some setting in the SF bug tracker that prevents people from attaching files to bug reports? Several people in #python-dev who aren't Python developers can't attach patches, but I'm not having any problems. --amk

4 5

RE: [Python-Dev] Re: various test failures - test_site, test_tempfile, test_bsddb/anydbm
by Skip Montanaro 06 Jun '04

06 Jun '04

>> It seems that the string posix_putenv() is passing to putenv() is >> properly allocated. Tim> It is, but "new" is embedded inside a Python string object: Tim> new = PyString_AS_STRING(newstr); Tim> A realloc(new) call would be insane (new isn't an address returned Tim> by malloc; newstr is the address that was returned by malloc). Agreed. I grabbed the Darwin libc source from Apple which corresponds to the version of Mac OS X I'm running. The only realloc() call in stdlib/setenv.c is to realloc the entire environment. The environment pointer is a funny beast. It starts out not in malloc'd storage but has to be malloc'd if it needs to be resized: static int alloced; /* if allocated space before */ ... char ***environp = _NSGetEnviron(); ... register int cnt; register char **p; for (p = *environp, cnt = 0; *p; ++p, ++cnt); if (alloced) { /* just increase size */ *environp = (char **)realloc((char *)*environp, (size_t)(sizeof(char *) * (cnt + 2))); if (!*environp) return (-1); } else { /* get new space */ alloced = 1; /* copy old entries into it */ p = malloc((size_t)(sizeof(char *) * (cnt + 2))); I stepped through the raw machine code in gdb. The alloced variable gets set properly (0 the first time through, then the space is malloc'd and alloced is set to 1). On the third call to setenv() the malloc_printf call is triggered. Tim> But if it's not trying to realloc(new), then new is a red herring, Tim> and the platform has gotten its pointers into the environment Tim> structure screwed up. That could be due to a wild store anywhere Tim> (platform, Python, whatever). *sigh* That seems to be the case. I haven't the energy to pick over machine code to find out where "anywhere" is... Skip

1 0