Mailman 3 October 2001 - Python-Dev

[development doc updates]
by Fred L. Drake Oct. 18, 2001

Oct. 18, 2001

The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Documentation as packaged with Python 2.2 beta 1.

1 0

2.2b1 release plan?
by Fredrik Lundh Oct. 18, 2001

Oct. 18, 2001

the PEP says October 10. I was planning to do some SRE work tonight, but I'm not sure I'll be able to squeeze it in. when is 2.2b1 going out? will I have time to work on this over the weekend? </F>

4 3

python2.1.1 SEGV in GC on Solaris 2.7
by Neil Schemenauer Oct. 18, 2001

Oct. 18, 2001

[I'm moving the discussion here from SF, using the tracker is too painful.] Anthony Baxter: > I've got a Zope installation where python2.1.1 is > segfaulting on Solaris2.7 - it's running a largish > ZEO server. [..] > Here's the trace with debugging enabled: > > #0 0xff00 in ?? () > #1 0x402f0 in collect (young=0x9b538, old=0x9b544) at > ./Modules/gcmodule.c:379 > #2 0x405a8 in collect_generations () at > ./Modules/gcmodule.c:484 > #3 0x40624 in _PyGC_Insert (op=0xbc1f24) at > ./Modules/gcmodule.c:507 > #4 0x5a224 in PyList_New (size=0) at Objects/listobject.c:61 > #5 0x21bc8 in eval_code2 (co=0x1cb370, globals=0x21bc0, > locals=0x67, > args=0x0, argcount=1, kws=0xf89b24, kwcount=0, defs=0x0, > defcount=0, > closure=0xbc1f24) at Python/ceval.c:1741 > > Next trick is to rebuild without any optimisation (sigh) > as I suspect that it's inlined subtract_refs(). Martin v. Löwis: > It would be interesting what the value of "gc" is at the > time of the crash. It looks like you got an object that > claims to support GC but has a null tp_traverse. Anthony Baxter: > Ok, I have an intact core file, and a matching binary, > no optimisations, nothing. This crash is showing the > crash at line 166 of gcmodule.c > traverse = PyObject_FROM_GC(gc)->ob_type->tp_traverse; > PyObject_FROM_GC(gc)->ob_type in this case is > > $24 = {ob_refcnt = 1, ob_type = 0x0} > > To check my logic, I checked gc_next and gc_prev using > the same GDB magic, and they correctly show up as a tuple > and an instance method. > > Some fiddling around seems to rule out stack space as the > problem, as well. We're going to try and see if purify > helps here, but the problem looks to be a junk object - > I have no idea how to track this down further. Help? > Would taking the horrible horrible hack of removing the > object from the gc linked list if ob_type is null help? > Well, it'd stop the crashes, anyway. Martin v. Löwis: > There are two options: > > a) the object isn't really a GC object, i.e. has no GC > header. In gdb, you can try to cast gc to PyObject*, then > see if the resulting pointer has a better ob_type (this is > unlikely, though, since the logic entering the object was > already using fromgc/togc) > > b) somebody has cleared the ob_type field. > > Can you guarantee that all extension modules have been > compiled with the 2.1.1 header files? > > Is the problem repeatable in the sense that gc will have the > same pointer value on each crash? If so, it is relatively > easy to track down: just set a gdb change watchpoint on the > address on the ob_type field of that address (note that > setting watchpoints is not possible until there is really > mapped memory on that address). > > If you can't analyse it through change breakpoints, I > recommend to annotate the interpreter in the following way: > in pyobject_init, put a printf that prints the address and > the tp_name of the type. In subtract_refs, if the ob_type > slot is null, print the address of the object and abort. > Then analyse the log to see whether a object really has been > allocated on that address, and what its type was (make sure > you consider the possibility that address are off by the > delta that FROM_GC adds). Anthony Baxter: > It's not a GC object. I'm positive all the extension > objects are correct - I just recompiled, without the > 1.5/2.0 headers around. > It's a different pointer each time round, unfortunately. It > also takes anything from 5 minutes to 2 hours to reproduce. > I've got about 4 copies of it running now, and I've got a > bunch of different core files. I've grabbed purify and an > eval license, and I'm feeding it the binary. > > The printf approach is probably not going to work - these > are busy busy Zope servers. Instead, my plan, assuming that > purify doesn't immediately spot a problem, is to change the > code so that if it gets a dud GC object, it will just bust > it out of the tree and let it leak, and print a message > saying so. Then I can quit the program, and purify will > tell me 'hey, you leaked!' and also tell me where it was > allocated. > > More concerning, about half the segfaults are not from the > GC at all, but from realloc in PyFrame_New (line 161 of > frameobject). These are the only two I'm getting - it's > split 50-50 amongst the 10 coredumps I have now. I'm not > sure whether to open a seperate bug for this. > > Has python2.1.1 been purified? With Zope and zope's > extensions? > > > Wow - it's amazing how this SF bug thing is so painful for > conversations :) The ob_type pointer must be getting cleared after the object has been added to the GC lists. The PyObject_IS_GC call in _PyGC_Insert would have segfaulted otherwise. Knowing the type of the object would be helpful in debugging the problem. I suggest reconsidering Martin's printf idea. You could add something like this to _PyGC_Insert: void _PyGC_Insert(PyObject *op) { static int did_open = 0; static FILE *log; if (!did_open) { did_open = 1; log = fopen("type.log", "w"); } fprintf(log, "%p %p\n", op, op->ob_type); ... Debugging this type of problem is really hard (as you already know) because the effect of the bug is found so far away from the source. Neil

4 4

warnings in dynload_shlib.c and posixmodule.c
by Guido van Rossum Oct. 18, 2001

Oct. 18, 2001

When compiling the latest CVS on Linux (Mandrake 8.0) I get these warnings: ../Python/dynload_shlib.c: In function `_PyImport_GetDynLoadFunc': ../Python/dynload_shlib.c:72: warning: implicit declaration of function `dlsym' ../Python/dynload_shlib.c:88: warning: implicit declaration of function `dlopen' ../Python/dynload_shlib.c:88: warning: assignment makes pointer from integer without a cast ../Python/dynload_shlib.c:91: warning: implicit declaration of function `dlerror' ../Python/dynload_shlib.c:91: warning: passing arg 2 of `PyErr_SetString' makes pointer from integer without a cast ../Modules/posixmodule.c: In function `posix_setgroups': ../Modules/posixmodule.c:3169: warning: implicit declaration of function `setgroups' Looks like some recent checkins... --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Function arguments
by Thomas Heller Oct. 18, 2001

Oct. 18, 2001

Call me weird, but sometimes I need functions taking a number of positional arguments, some named arguments with default values, and other named arguments as well. Something like this (which does _not_ work): def function(*args, defarg1=None, defarg2=0, **kw): ... The usual way to program this is: def function(*args, ***kw): if kw.has_key('defarg1'): defarg1 = kw['defarg1'] del kw['defarg1'] else: defarg1 = None if kw.has_key('defarg2'): defarg1 = kw['defarg2'] del kw['defarg2'] else: defarg2 = 0 # ...process the positional arguments in args # ...process the remaining named arguments in kw Doesn't look very pretty IMO, and using the dictionaries' get method doesn't help. I was thinking of a popitem() dictionary method taking (optionally) 2 arguments: the name of the item to pop, and the default value to return if the item is not present in the dictionary: >>> d = {'a': 2, 'b': 3} >>> d.popitem('defarg', 0) 0 >>> d {'a': 2, 'b': 3} >>> d.popitem('a', 100) 2 >>> d {'b': 3} >>> Opinions? Thomas

3 4

Re: python2.1.1 SEGV in GC on Solaris 2.7
by Anthony Baxter Oct. 18, 2001

Oct. 18, 2001

>>> Neil Schemenauer wrote > [I'm moving the discussion here from SF, using the tracker is too > painful.] ok. please CC me on any replies. current update - I applied the PyFrame_BlockSetup() fix for ceval.c that Martin suggested, and so far I've had 4 processes running for a couple of hours each, without any splattage. this doesn't necessarily mean it's fixed - the time between failures has varied from 5 minutes to 12 hours over the last few days. > The ob_type pointer must be getting cleared after the object has been > added to the GC lists. The PyObject_IS_GC call in _PyGC_Insert would > have segfaulted otherwise. Knowing the type of the object would be > helpful in debugging the problem. I suggest reconsidering Martin's > printf idea. You could add something like this to _PyGC_Insert: I'll have a poke at this tomorrow, and see if it's possible at all - the problem is that the boxes are part of a zope cluster that gets a _lot_ of hits (something over 1m/day) so that's a lot of objects... I'm also going to have another go at getting purify to work on our stripped down production servers. > Debugging this type of problem is really hard (as you already know) > because the effect of the bug is found so far away from the source. *nod* I'm aware of it. thanks! Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

1 0

Re: [Python-Dev] more pychecker warnings from python-current
by Michael Hudson Oct. 18, 2001

Oct. 18, 2001

"Tim Peters" <tim.one(a)home.com> writes: > I wouldn't be averse to adding, e.g., > > if 0: > prefix # make prefix appear used to checking tools > > but sooner or later checking tools will complain about that too. If they work by scanning bytecode (as PyChecker does, doesn't it?), that won't work today. pointless-ly y'rs, M. -- Screaming 14-year-old boys attempting to prove to each other that they are more 3133t than j00. -- Reason #8 for quitting slashdot today, from http://www.cs.washington.edu/homes/klee/misc/slashdot.html

1 0

more pychecker warnings from python-current
by Neal Norwitz Oct. 18, 2001

Oct. 18, 2001

Well, almost python-current. The warnings were from a few days ago. Neal -- cgi.py:880: Local variable (traceback) not used copy_reg.py:47: self is argument in function not sure if this is really a problem or intended copy_reg.py:65: Function return types are inconsistent line 63 returns tuple of 3, but line 65, returns tuple of 2 doctest.py:528: Parameter (prefix) not used docstring says prefix is used, but it isn't locale.py:358: Redefining attribute (setlocale) original line (68) markupbase.py:31: No class attribute (rawdata) found markupbase.py:49: Local variable (sys) not used markupbase.py:71: No class attribute (unknown_decl) found markupbase.py:86: No class attribute (error) found markupbase.py:118: No class attribute (parse_comment) found markupbase.py:166: Local variable (n) not used mhlib.py:651: Local variable (messages) not used looks like a return in the if clause is necessary, otherwise exception pickle.py:607: No class attribute (persistent_load) found pickle.py:616: No class attribute (persistent_load) found profile.py:356: Local variable (a) not used on next line, return is not necessary, should a be returned? profile.py:368: Local variable (a) not used xmlrpclib.py:1: Imported module (sys) not used

4 4

re: conditional expressions
by Greg Wilson Oct. 18, 2001

Oct. 18, 2001

I just showed the proposed conditional expression syntax to two colleagues, and asked them how they would write the "sign" function (-1 for negative, 0 for zero, +1 for positive) using it. Both of them wrote: s = if x < 0 then -1 elif x == 0 then 0 else 1 (Well, OK, one of them tested negative, then positive, then returned zero as the 'else', but that's a quibble.) I think most users will expect 'elif' to work in expressions, just as it does in statements; is this already part of the new syntax? Thanks, Greg ----------------------------------------------------------------------------------------------------------------- The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Baltimore Technologies plc will not be liable for direct, special, indirect or consequential damages arising from alteration of the contents of this message by a third party or as a result of any virus being passed on. In addition, certain Marketing collateral may be added from time to time to promote Baltimore Technologies products, services, Global e-Security or appearance at trade shows and conferences. This footnote confirms that this email message has been swept by Baltimore MIMEsweeper for Content Security threats, including computer viruses.

2 1

bug or feature in imp.load_module()?
by Neal Norwitz Oct. 17, 2001

Oct. 17, 2001

module = imp.load_module(path, file, filename, smt) After returning from imp.load_module(), the file paramter passed in may be at the beginning or end depending on whether it was byte compiled. If the file was already compiled, the file returned will stay at the beginning. If the file was byte compiled in load_module(), the file will be at the end. The following program may demonstrate better: ##################### import imp, sys, os FILE_BASE = 'x' PY_FILE = FILE_BASE + '.py' def load(f): file, filename, smt = imp.find_module(f, sys.path) imp.load_module('', file, filename, smt) print 'File is at byte:', file.tell() def main(): try: os.unlink(PY_FILE + 'c') except: pass if not os.path.exists(PY_FILE): file = open(PY_FILE, 'w') file.close() load(FILE_BASE) load(FILE_BASE) if __name__ == '__main__': main() ##################### file sometimes being modified was unexpected. And I couldn't find the behaviour documented anywhere. This behaviour goes back to Python 1.5 at least AFAIR. Neal

2 4