Mailman 3 June 2002 - Python-Dev

Null checking
by David Abrahams 10 Jun '02

10 Jun '02

A couple of quick questions for the authors of the Python source: I notice that most, if not all, of the Python 'C' API includes null checks for the PyObject* arguments, meaning that you can't crash Python by passing the result of a previous operation, even if it returns an error. First question: can that be counted on? Hmm, I guess I've answered my own question -- PyNumber_InPlaceAdd has no checks. I note that the null_error() check in abstract.c is non-destructive: it preserves any existing error, whereas other checks (e.g. in typeobject.c) do not. Second question: I guess I really want to know what the intention behind these checks is. Is it something like "prevent extension writers from crashing Python in some large percentage of cases", or is there a deeper plan that I'm missing? TIA, Dave +---------------------------------------------------------------+ David Abrahams C++ Booster (http://www.boost.org) O__ == Pythonista (http://www.python.org) c/ /'_ == resume: http://users.rcn.com/abrahams/resume.html (*) \(*) == email: david.abrahams(a)rcn.com +---------------------------------------------------------------+

4 7

Re: [Python-Dev] Bizarre new test failure
by Guido van Rossum 10 Jun '02

10 Jun '02

Offline, Jeremy, Neil, Tim & I figured out what was really the case here. I believe the last thing we reported here was that we'd found a sample program that required several gc.collect() calls to clean out all its garbage, which surprised Neil and Tim. Since they know the collector inside out, if it surprises them, there's probably a bug. Here's the analysis. A new-style instance increfs its class (its ob_type), to make sure that the class doesn't go away as long as the instance exists. But the tp_traverse handler for new-style instances didn't visit the ob_type reference, so the collector thinks there are "outside" references to the class, and doesn't collect it. When the last instance goes away, the refcnt to the class finally matches what the collector sees, and it can collect the class (once all other references to it are gone of course). I was tempted to say "this ain't so bad, let's keep it that way." But Jeremy and Tim came up with counterexamples involving a cycle between a class and its instance, where the cycle would never be broken. Tim found that this program grows without bounds, though ever slower, since it spends more and more time in the 2nd generation collector, where all the uncollected objects eventually end up: while 1: class A(object): pass A.a = A() I tried the simplest possible fix, which was to visit self->ob_type in the new-style instance tp_traverse handler (subtype_traverse() in typeobject.c). But this caused assertions to fail all over the place. It turns out that when the collector decides to break a cycle like this, it calls the tp_clear handler for each object in the cycle, and then the subsequent deletion of the instance references the type in ways that have been made invalid by the clearing of the type. So this was a dead end. So here's a patch that does do the right thing. It adds a new field to type objects, tp_dependents. This is essentially a second reference count, counting the instances and direct subclasses. As long as tp_dependents is nonzero, this means there are still instances or subclasses, and then the type's tp_clear handler doesn't do anything. A consequence of the patch is that there will always be examples that take more than one collection to clean their garbage -- but eventually all garbage will be cleared out. (I suppose a worst-case example would be a very long chain of subclasses, which would be cleared out once class per collection.) Consequently, I'm patching test_gc.py to get rid of garbage left behind by previous tests in a loop. A downside of the patch is that it adds a new field to the type object structure; I believe this prevents it from being a backport candidate to 2.2.2. For a half blissful hour I believed that it would be possible to do something much simpler by doubling the regular refcnt; then I realized that the double refcnt would mean the collector would never break the cycle. Still, I am wishing for a solution that avoids adding a new field. Please comment on this patch! Index: Include/object.h =================================================================== RCS file: /cvsroot/python/python/dist/src/Include/object.h,v retrieving revision 2.101 diff -c -r2.101 object.h *** Include/object.h 12 Apr 2002 01:57:06 -0000 2.101 --- Include/object.h 8 Jun 2002 13:09:14 -0000 *************** *** 292,297 **** --- 292,298 ---- PyObject *tp_cache; PyObject *tp_subclasses; PyObject *tp_weaklist; + int tp_dependents; #ifdef COUNT_ALLOCS /* these must be last and never explicitly initialized */ Index: Objects/typeobject.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Objects/typeobject.c,v retrieving revision 2.148 diff -c -r2.148 typeobject.c *** Objects/typeobject.c 4 Jun 2002 19:52:53 -0000 2.148 --- Objects/typeobject.c 8 Jun 2002 13:09:15 -0000 *************** *** 218,225 **** memset(obj, '\0', size); ! if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) Py_INCREF(type); if (type->tp_itemsize == 0) PyObject_INIT(obj, type); --- 218,227 ---- memset(obj, '\0', size); ! if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) { Py_INCREF(type); + type->tp_dependents++; + } if (type->tp_itemsize == 0) PyObject_INIT(obj, type); *************** *** 290,295 **** --- 292,303 ---- } } + if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) { + int err = visit((PyObject *)type, arg); + if (err) + return err; + } + if (basetraverse) return basetraverse(self, visit, arg); return 0; *************** *** 464,469 **** --- 472,478 ---- /* Can't reference self beyond this point */ if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) { Py_DECREF(type); + type->tp_dependents--; } } *************** *** 1170,1175 **** --- 1179,1192 ---- Py_INCREF(base); type->tp_base = base; + /* Incref the bases' tp_dependents count */ + for (i = 0; i < nbases; i++) { + PyTypeObject *b; + b = (PyTypeObject *)PyTuple_GET_ITEM(bases, i); + if (PyType_Check(b) && (b->tp_flags & Py_TPFLAGS_HEAPTYPE)) + b->tp_dependents++; + } + /* Initialize tp_dict from passed-in dict */ type->tp_dict = dict = PyDict_Copy(dict); if (dict == NULL) { *************** *** 1431,1441 **** --- 1448,1475 ---- static void type_dealloc(PyTypeObject *type) { + PyObject *bases; etype *et; /* Assert this is a heap-allocated type object */ assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE); _PyObject_GC_UNTRACK(type); + + /* Decref the bases' tp_dependents count */ + bases = type->tp_bases; + if (bases) { + int i, nbases; + assert(PyTuple_Check(bases)); + nbases = PyTuple_GET_SIZE(bases); + for (i = 0; i < nbases; i++) { + PyTypeObject *b; + b = (PyTypeObject *)PyTuple_GET_ITEM(bases, i); + if (PyType_Check(b) && + (b->tp_flags & Py_TPFLAGS_HEAPTYPE)) + b->tp_dependents--; + } + } + PyObject_ClearWeakRefs((PyObject *)type); et = (etype *)type; Py_XDECREF(type->tp_base); *************** *** 1495,1502 **** etype *et; int err; ! if (!(type->tp_flags & Py_TPFLAGS_HEAPTYPE)) ! return 0; et = (etype *)type; --- 1529,1535 ---- etype *et; int err; ! assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE); et = (etype *)type; *************** *** 1524,1533 **** type_clear(PyTypeObject *type) { etype *et; ! PyObject *tmp; ! if (!(type->tp_flags & Py_TPFLAGS_HEAPTYPE)) ! return 0; et = (etype *)type; --- 1557,1583 ---- type_clear(PyTypeObject *type) { etype *et; ! PyObject *tmp, *bases; ! assert(type->tp_flags & Py_TPFLAGS_HEAPTYPE); ! ! if (type->tp_dependents) ! return 0; /* Not yet, there are still instances */ ! ! /* Decref the bases' tp_dependents count */ ! bases = type->tp_bases; ! if (bases) { ! int i, nbases; ! assert(PyTuple_Check(bases)); ! nbases = PyTuple_GET_SIZE(bases); ! for (i = 0; i < nbases; i++) { ! PyTypeObject *b; ! b = (PyTypeObject *)PyTuple_GET_ITEM(bases, i); ! if (PyType_Check(b) && ! (b->tp_flags & Py_TPFLAGS_HEAPTYPE)) ! b->tp_dependents--; ! } ! } et = (etype *)type; *************** *** 1754,1763 **** --- 1804,1815 ---- } if (new->tp_flags & Py_TPFLAGS_HEAPTYPE) { Py_INCREF(new); + new->tp_dependents++; } self->ob_type = new; if (old->tp_flags & Py_TPFLAGS_HEAPTYPE) { Py_DECREF(old); + old->tp_dependents--; } return 0; } Index: Lib/test/test_gc.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_gc.py,v retrieving revision 1.14 diff -c -r1.14 test_gc.py *** Lib/test/test_gc.py 28 Mar 2002 21:22:25 -0000 1.14 --- Lib/test/test_gc.py 8 Jun 2002 13:09:15 -0000 *************** *** 220,228 **** def test(): if verbose: print "disabling automatic collection" enabled = gc.isenabled() gc.disable() ! verify(not gc.isenabled() ) debug = gc.get_debug() gc.set_debug(debug & ~gc.DEBUG_LEAK) # this test is supposed to leak --- 220,229 ---- def test(): if verbose: print "disabling automatic collection" + while gc.collect(): pass # collect garbage from previous tests enabled = gc.isenabled() gc.disable() ! verify(not gc.isenabled()) debug = gc.get_debug() gc.set_debug(debug & ~gc.DEBUG_LEAK) # this test is supposed to leak --Guido van Rossum (home page: http://www.python.org/~guido/)

3 8

Re: [Python-Dev] Null checking
by Michael Hudson 10 Jun '02

10 Jun '02

"David Abrahams" <david.abrahams(a)rcn.com> writes: > A couple of quick questions for the authors of the Python source: I notice > that most, if not all, of the Python 'C' API includes null checks for the > PyObject* arguments, meaning that you can't crash Python by passing the > result of a previous operation, even if it returns an error. > > First question: can that be counted on? Hmm, I guess I've answered my own > question -- PyNumber_InPlaceAdd has no checks. You got it. > I note that the null_error() check in abstract.c is non-destructive: it > preserves any existing error, whereas other checks (e.g. in typeobject.c) > do not. > > Second question: I guess I really want to know what the intention behind > these checks is. I'm not sure there is one. It may just be a bad example of defensive programming (cf. OOSC). > Is it something like "prevent extension writers from crashing Python > in some large percentage of cases", or is there a deeper plan that > I'm missing? Well, if you're missing it, so am I. I'd also like to know why all the (for instance) methods in tupleobject.c start with "if (!PyTuple_Check(self)". You'd have to try REALLY hard to get those tests to fail... Cheers, M. -- Q: What are 1000 lawyers at the bottom of the ocean? A: A good start. (A lawyer told me this joke.) -- Michael Ströder, comp.lang.python

3 2

Re: [Python-Dev] Null checking
by Michael Hudson 10 Jun '02

10 Jun '02

Guido van Rossum <guido(a)python.org> writes: > > I'd also like to know why all the (for instance) methods in > > tupleobject.c start with "if (!PyTuple_Check(self)". You'd have to > > try REALLY hard to get those tests to fail... > > I found exactly one call to PyTuple_Check() that satisfies that > description, and it was in your own (uncommitted) addition, > tuplesubscript(). :-) Yeah. I wonder what I was thinking when I wrote that (it was two years ago now, after all). Never mind, I'll do my research better before my next post. Cheers, M. -- GET *BONK* BACK *BONK* IN *BONK* THERE *BONK* -- Naich using the troll hammer in cam.misc

1 0

os.stat(filename).r_dev
by Gustavo Niemeyer 10 Jun '02

10 Jun '02

While talking to Lars about tarfile.py, he has noted an interesting detail in the current implementation of os.stat(filename).r_dev: ------------- With the current os.stat(), it is impossible to implement addition of those devices. That's because the result for st_rdev is just a plain integer, which still must be divided into the major and minor part. This division (resp. the C type dev_t) differs between several operating systems: OS format major minor Linux 32-bit upper 16bits lower 16bits SVR4 32-bit upper 14bits lower 18bits BSD 16-bit upper 8bits lower 8bits ------------- It seems like we really need some way to decode r_dev. One possible solutions are to implement major(), minor(), and makedev() somewhere. Another solution, if r_dev's raw value has no obvious use, would be to turn it into a two elements tuple like (major, minor). Any suggestions? -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ]

3 5

pymemcompat.h & PyMem_New and friends
by Michael Hudson 10 Jun '02

10 Jun '02

Michael Hudson <mwh(a)python.net> writes: > /* There are three "families" of memory API: the "raw memory", "object > memory" and "object" families. (This is ignoring the matter of the > cycle collector, about which more is said below). Of course this is an over-simplification. There is at least one other family in fairly widespread use in the Python core; the "typed memory allocator", PyMem_New, PyMem_Resize and PyMem_Del. Should this family be listed in pyemcompat.h or subtly discouraged? (I don't think there are any other options). I think it should be subtly discouraged, for a couple of reasons: a) three is a smaller number than four. b) there is a non-analogy: PyMem_Malloc ---> PyMem_New PyObject_Malloc ---> PyObject_New They do rather different things. c) I don't think omitting a cast and a sizeof is that much of a win. I'm not proposing actually taking these interfaces away. (as a special bonus I won't even mention the fact that we have PyMem_Resize, PyObject_GC_Resize (only used in listobject.c) but not PyObject_Resize...) Cheers, M. -- If a train station is a place where a train stops, what's a workstation? -- unknown (to me, at least)

2 1

Subclassing threading.Thread
by Bernard Yue 09 Jun '02

09 Jun '02

Attached files test1.py and test2.py produce different result. Is it a bug? Bernie #!/usr/bin/env python import socket import threading class server: def __init__(self): self._addr_local = ('127.0.0.1', 25339) self._s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self._s.bind(self._addr_local) def test(): for i in range(10): a = threading.Thread( target=server) a.start() a.run() a.join() if __name__ == '__main__': test() #!/usr/bin/env python import socket import threading class server(threading.Thread): def __init__(self): threading.Thread.__init__(self) self.__addr_local = ('127.0.0.1', 25339) self.__s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.__s.bind(self.__addr_local) def test(): for i in range(10): a = server() a.start() a.run() a.join() if __name__ == '__main__': test()

2 1

Changing ob_size to [s]size_t
by Perry Greenfield 08 Jun '02

08 Jun '02

Guido writes: > I'm not very concerned about strings or lists with more than 2GB > items, but I am concerned about other memory buffers. Those in the Numeric/numarray community, for one, would also be concerned. Although there aren't many data arrays these days that are larger than 2GB there are some beginning to appear. I have no doubt that within a few years there will be many more. I'm not sure I understand all the implications of the discussion here, but it sounds like an important issue. Currently strings are frequently used as a common "medium" to pass binary data from one module to another (e.g., from Numeric to PIL); limiting strings to 2GB may prove a problem in this area (though frankly, I suspect few will want to use them as temporary buffers for objects that size until memories have grown a bit more :-). Perry Greenfield

5 9

Bizarre new test failure
by Tim Peters 07 Jun '02

07 Jun '02

Guido noticed this on Linux late this afternoon. I've seen it on Win2K and Win98SE since. test_gc fails if you run the whole test suite from the start: test test_gc failed -- test_list: actual 10, expected 1 This seems impossible (look at the test). It doesn't fail in isolation. It fails in both debug and release builds. Not *all* tests before test_gc need to be run in order to provoke a failure, but I can detect no sense in which do need to be run (not just one or two, but lots of them). Here are the files that changed between a Python that does work (yesterday) and now: P python/configure P python/configure.in P python/pyconfig.h.in P python/Doc/lib/libgetopt.tex P python/Doc/lib/libsocket.tex P python/Lib/copy.py P python/Lib/fileinput.py P python/Lib/getopt.py P python/Lib/posixpath.py P python/Lib/shutil.py P python/Lib/socket.py P python/Lib/compiler/pyassem.py P python/Lib/compiler/pycodegen.py P python/Lib/compiler/transformer.py P python/Lib/distutils/command/clean.py P python/Lib/test/test_b1.py P python/Lib/test/test_commands.py P python/Lib/test/test_descr.py P python/Lib/test/test_getopt.py P python/Lib/test/test_socket.py U python/Lib/test/test_timeout.py P python/Misc/ACKS P python/Misc/NEWS P python/Modules/gcmodule.c P python/Modules/socketmodule.c P python/Modules/socketmodule.h P python/Objects/abstract.c P python/Objects/complexobject.c P python/Objects/rangeobject.c P python/Tools/webchecker/webchecker.py While Jeremy did fiddle gcmodule.c, that isn't the cause. I changed the test like so: def test_list(): import sys l = [] l.append(l) gc.collect() del l gc.set_debug(gc.DEBUG_SAVEALL) n = gc.collect() print >> sys.stderr, '*' * 30, n, gc.garbage expect(n, 1, "list") Here's the list of garbage objects it found: [<class 'test_descr.mysuper'>, {'__dict__': <attribute '__dict__' of 'mysuper' objects>, '__module__': 'test_descr', '__weakref__': <member '__weakref__' of 'mysuper' objects>, '__doc__': None, '__init__': <function __init__ at 0x00CC00D8>}, (<class 'test_descr.mysuper'>, <type 'super'>, <type 'object'>), (<type 'super'>,), [[...]], <attribute '__dict__' of 'mysuper' objects>, <member '__weakref__' of 'mysuper' objects>, <function __init__ at 0x00CC00D8>, (<cell at 0x00CB4DB0: type object at 0x00768420>,), <cell at 0x00CB4DB0: type object at 0x00768420>] The recursive list: [[...]] is the only one expected here. Your turn <wink>.

8 21

d.get_key(key) -> key?
by Daniel 'eikeon' Krech 07 Jun '02

07 Jun '02

While attempting to "intern" the nodes in our rdflib's triple store I have come across the following question. Is there or could there be an efficient way to get an existing key from a dictionary given a key that is == but whose id is not. For example: given a==b and id(a)!=id(b) and d[a] = 1 what is the best way to: d.get_key(b) -> a --eikeon, http://eikeon.com/ PS: Here is the code where I am trying to get rid of multiple instances of equivalent nodes: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/redfoot/rdflib-1.0/rdflib/st… and a not so efficient first attempt: # (could use a s.get(e2)->e1 as well given e1==e2 and id(e1)!=id(e2)) class Set(object): def __init__(self): self.__set = [] def add(self, obj): e = self.get(obj) if e: # already have equivalent element, so return the one we have return e else: self.__set.append(obj) return obj def get(self, obj): if obj in self.__set: for e in self.__set: if e==obj: return e return None class Intern(object): def __init__(self): super(Intern, self).__init__() self.__nodes = Set() def add(self, subject, predicate, object): subject = self.__nodes.add(subject) predicate = self.__nodes.add(predicate) object = self.__nodes.add(object) super(Intern, self).add(subject, predicate, object)

7 29