Mailman 3 February 2002 - Python-Dev

Global name lookup schemes
by Ka-Ping Yee Feb. 25, 2002

Feb. 25, 2002

Okay, i spent another afternoon drawing silly pictures full of boxes and arrows. I swear, i'm going to be seeing pointers in my dreams tonight. Here are figures representing my current understanding of the various schemes on the table: Jeremy 1: the dlict scheme http://lfw.org/python/jeremy1.gif http://lfw.org/python/jeremy1.tif http://lfw.org/python/jeremy1.ai Jeremy, i think i'm still somewhat unclear -- notice the two question marks in the figure. What kind of animal is the cache? I assumed that the invalidation info lives in an array parallel to the dlict's array. Is this right? Guido 1: the original cellptr/objptr scheme http://lfw.org/python/guido1.gif http://lfw.org/python/guido1.tif http://lfw.org/python/guido1.ai Ping 1: guido1 + a tweak to always use two dereferencing steps http://lfw.org/python/ping1.gif http://lfw.org/python/ping1.tif http://lfw.org/python/ping1.ai Tim 1: timdicts and cells with shadow flags http://lfw.org/python/tim1.gif http://lfw.org/python/tim1.tif http://lfw.org/python/tim1.ai GIFs are small versions, TIFs are big versions, AIs are Adobe Illustrator source files. Please examine, send me corrections, discuss, enjoy... :) Still to do: Guido 2: the globals_vector scheme Skip 1: the global-tracking scheme (I don't actually know yet what in this diagram would be different from the way things work today. Statically, Skip's picture is mostly the same; it's the runtime behaviour that's different. Still, it's probably good to have a reference picture of today's data structures anyway.) -- ?!ng

10 22

Pthreads wizard needed to look at bug #522393
by Jack Jansen Feb. 25, 2002

Feb. 25, 2002

Folks, in the process of testing 2.2.1 I ran into a bug on SGI that has been there since at least 2.2: if you build with threads you get an undefined error on pthread_detach() while linking the interpreter. I think the solution is to change the autoconf test that decides which libraries to add for pthread (make it refer not only the thread_create() but also thread_detach()), but as pthread_detach apparently isn't available in all pthread implementations (if I understand the small forest of ifdefs inside thread_pthread.h correctly) I'm not sure this won't break anything else. Could some pthread guru please have a look at bug #522393? -- - Jack Jansen <Jack.Jansen(a)oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

1 0

Webhosting en PESOS!
by info＠virtucomnetworks.com Feb. 24, 2002

Feb. 24, 2002

1 0

A little GC confusion
by David Abrahams Feb. 23, 2002

Feb. 23, 2002

Hi All, The following extension module (AA) is a reduced example of what I'm doing to make extension classes in 2.2. I followed the examples given by typeobject.c. When I "import AA,pdb" I get a crash in GC. Investigating further, I see this makes sense: GC is enabled in class_metatype_object, yet class_type_object does not follow the first rule of objects whose type has GC enabled: "The memory for the object must be allocated using PyObject_GC_New() or PyObject_GC_VarNew()." So, I guess the question is, how does PyBaseObject_Type (also statically allocated) get away with it? TIA, Dave ---------------- // Copyright David Abrahams 2002. Permission to copy, use, // modify, sell and distribute this software is granted provided this // copyright notice appears in all copies. This software is provided // "as is" without express or implied warranty, and with no claim as // to its suitability for any purpose. #include <Python.h> PyTypeObject class_metatype_object = { PyObject_HEAD_INIT(0) 0, "Boost.Python.class", PyType_Type.tp_basicsize, 0, 0, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, // &PyType_Type, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, // PyType_GenericNew /* tp_new */ }; // Get the metatype object for all extension classes. PyObject* class_metatype() { if (class_metatype_object.tp_dict == 0) { class_metatype_object.ob_type = &PyType_Type; class_metatype_object.tp_base = &PyType_Type; if (PyType_Ready(&class_metatype_object)) return 0; } Py_INCREF(&class_metatype_object); return (PyObject*)&class_metatype_object; } // Each extension instance will be one of these typedef struct instance { PyObject_HEAD void* objects; } instance; static void instance_dealloc(PyObject* inst) { instance* kill_me = (instance*)inst; inst->ob_type->tp_free(inst); } PyTypeObject class_type_object = { PyObject_HEAD_INIT(0) file://&class_metatype_object) 0, "Boost.Python.instance", sizeof(PyObject), 0, instance_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, file://&PyBaseObject_Type, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ PyType_GenericAlloc, /* tp_alloc */ PyType_GenericNew }; PyObject* class_type() { if (class_type_object.tp_dict == 0) { class_type_object.ob_type = (PyTypeObject*)class_metatype(); class_type_object.tp_base = &PyBaseObject_Type; if (PyType_Ready(&class_type_object)) return 0; } Py_INCREF(&class_type_object); return (PyObject*)&class_type_object; } PyObject* make_class() { PyObject* bases, *args, *mt, *result; bases = PyTuple_New(1); PyTuple_SET_ITEM(bases, 0, class_type()); args = PyTuple_New(3); PyTuple_SET_ITEM(args, 0, PyString_FromString("AA")); PyTuple_SET_ITEM(args, 1, bases); PyTuple_SET_ITEM(args, 2, PyDict_New()); mt = class_metatype(); result = PyObject_CallObject(mt, args); Py_XDECREF(mt); Py_XDECREF(args); return result; } static PyMethodDef SpamMethods[] = { {NULL, NULL} /* Sentinel */ }; DL_EXPORT(void) initAA() { PyObject *m, *d; m = Py_InitModule("AA", SpamMethods); d = PyModule_GetDict(m); PyDict_SetItemString(d, "AA", make_class()); } +---------------------------------------------------------------+ David Abrahams C++ Booster (http://www.boost.org) O__ == Pythonista (http://www.python.org) c/ /'_ == resume: http://users.rcn.com/abrahams/resume.html (*) \(*) == email: david.abrahams(a)rcn.com +---------------------------------------------------------------+

9 19

Re: [Python-Dev] What's blocking 2.2.1?
by Michael Hudson Feb. 23, 2002

Feb. 23, 2002

Tim Peters <tim.one(a)comcast.net> writes: > [Michael Hudson] > > I think I'm caught up on porting fixes from the trunk to the > > release22-maint branch. > > Voluminous thanks for the work, Michael! It's now *really* easy for me to get checkins across to the branch. Does anyone else use gnus to read their -checkins mail? I could probably put the little bundle of scripts I use under Tools/scripts/. Cheers, M. -- In short, just business as usual in the wacky world of floating point <wink>. -- Tim Peters, comp.lang.python

1 0

Re: [Python-Dev] What's blocking 2.2.1?
by Michael Hudson Feb. 23, 2002

Feb. 23, 2002

Jack Jansen <Jack.Jansen(a)oratrix.com> writes: > On Friday, February 22, 2002, at 03:10 , Michael Hudson wrote: > > > I think I'm caught up on porting fixes from the trunk to the > > release22-maint branch. Now would be a good time to shout if you > > think I've missed something (although I might not read my email before > > Monday). > > > > (I've entirely ignored the Mac subtree here. Jack, that's your > > problem, I'm afraid). > > I'll do a quick check to see whether there's anything that is vital for > Mac OS X unix Python that has to go in, I'll let you know. Fine. I've done the first one. > I think MacPython will have to be done after the unix/win distribution > has been made, but that depends on your timeframe (i.e. if you can hand > the Mac/ portion of the tree over to me real soon now I can try and > squeeze the time in). ? Sorry, you've lost me. I'm afraid. The Mac/ portion of the tree is yours. Cheers, M. -- That's why the smartest companies use Common Lisp, but lie about it so all their competitors think Lisp is slow and C++ is fast. (This rumor has, however, gotten a little out of hand. :) -- Erik Naggum, comp.lang.lisp

1 0

What's blocking 2.2.1?
by Michael Hudson Feb. 23, 2002

Feb. 23, 2002

I think I'm caught up on porting fixes from the trunk to the release22-maint branch. Now would be a good time to shout if you think I've missed something (although I might not read my email before Monday). (I've entirely ignored the Mac subtree here. Jack, that's your problem, I'm afraid). There are still some bugs in the trunk that need fixing, though. [ #496873 ] cPickle / time.struct_time loop - I think this one is firmly in Guido's domain. [ #501591 ] dir() doc is old - probably not that hard. are all that are marked as 2.2.1 candidates (apart from two MacOS bugs), but there are probably more. I don't want to trawl through all the 250+ (!) open bugs to look for them if I don't have to, so can I ask people to nominate bugs they know of? Cheers, M. -- We've had a lot of problems going from glibc 2.0 to glibc 2.1. People claim binary compatibility. Except for functions they don't like. -- Peter Van Eynde, comp.lang.lisp

4 3

Re: [Python-Dev] 2.2.1 issues
by Michael Hudson Feb. 23, 2002

Feb. 23, 2002

martin(a)v.loewis.de (Martin v. Loewis) writes: > "M.-A. Lemburg" <mal(a)lemburg.com> writes: > > > Right. 1) was caused by 2). > > That wasn't actually the case. The overwriting of memory was really > independent of the error in surrogate processing, and can be fixed > independently. OK, thanks for the clarification. > > As a result, modules using unpaired surrogates in Unicode > > literals are simply broken in Python <= 2.2.0. > > I think this is unimportant enough to just accept this bug for Python > 2.2.x. If people ever run into the problem, well: just don't do this. > Unpaired surrogates will be entirely in Unicode 3.2. I think you're missing a word in the last sentence? > > The problem with backporting this patch is that in order > > for Python to properly recompile any broken module, the > > magic will have to be changed. Question is whether this > > is a reasonable thing to do in a patch level release... > > The memory-overwriting problem can be fixed independently, e.g. with > > https://sourceforge.net/tracker/download.php?group_id=5470&atid=105470&file… Thanks, I've now checked this fix in, and will consider the whole issue to be closed until further notice. Cheers, M. -- That's why the smartest companies use Common Lisp, but lie about it so all their competitors think Lisp is slow and C++ is fast. (This rumor has, however, gotten a little out of hand. :) -- Erik Naggum, comp.lang.lisp

2 1

patch: speed up name access by up to 80%
by Oren Tirosh Feb. 22, 2002

Feb. 22, 2002

Problem: Python name lookup in dictionaries is relatively slow. Possible solutions: 1. Find ways to bypass dictionary lookup. 2. Optimize the hell out of dictionary lookup. Examples of approach #1 are the existing fastlocals mechanism, PEP 266, PEP 267 and the recent proposal by GvR for speeding up instance attribute access. These proposals all face major difficulties resulting from the dynamic nature of Python's namespace and require tricky techniques of code analysis code to check for assignments that may change the visible namespace. I have chosen to try #2. The biggest difficulty with this approach is that dictionaries are already very well optimized :-) But I have found that Python dictionaries are mostly optimized for general-purpose use, not for use as namespace for running code. There are some characteristics unique to namespace access which have not been used for optimization: * Lookup keys are always interned strings. * The local/global/builtin fallback means that most accesses fail. The patch adds the function PyDict_GetItem_Fast to dictobject. This function is equivalent to PyDict_GetItem but is much faster for lookup using interned string keys and for lookups with a negative result. LOAD_NAME and LOAD_GLOBAL have been converted to use this function. Here are the timings for 200,000,000 accesses to fastlocals, locals, globals and builtins for Python 2.2 with and without the fastnames patch: 2.2 2.2+fastnames --------------------------------------- builtin.py 68.540s 37.900s global.py 54.570s 34.020s local.py 41.210s 32.780s fastlocal.py 24.530s 24.540s Fastlocals are still significantly faster than locals, but not by such a wide margin. You can notice that the speed differences between locals, globals and builtins are almost gone. The machine is a Pentium III at 866MHz running linux. Both interpreters were compiled with "./configure ; make". The test code is at the bottom of this message. You will not see any effect on the results of pybench because it uses only fastlocals inside the test loops. Real code that uses a lot of globals and builtins should see a significant improvement. Get the patch: http://www.tothink.com/python/fastnames/fastnames.patch The optimization techniques used: * Inline code PyDict_GetItem_Fast is actually an inline macro. If the item is stored at the first hash location it will be returned without any function calls. It also requires that the key used for the entry is the same interned string as the key. This fast inline version is possible because the first argument is known to be a valid dictionary and the second argument is known to be a valid interned string with a valid cached hash. * Key comparison by pointer If the inline macro fails PyDict_GetItem_Fast2 is called. This function searches for an entry with a key identical to the requested key. This search is faster than lookdict or lookdict_string because there are no expensive calls to external compare-by-value functions. Another small speedup is gained by not checking for free slots since this function is never used for setting items. If this search fails, the dictionary's ma_lookup function is called. * Interning of entry keys One reason the quick search could fail is because the entry was set using direct access to __dict__ instead of standard name assignment and therefore the entry key is not interned. In this case the entry key is replaced with the interned lookup key. The next fast search for the same key will succeed. There is a very good chance that it will be handled by the inline macro. * Negative entries In name lookup most accesses fail. In order to speed them up negative entries can mark a name as "positively not there", usually detected by the macro without requiring any function calls. Negative entries have the interned key as their me_key and me_value is NULL. Negative entries occupy real space in the hash table and cannot be reused as empty slots. This optimization technique is not practical for general purpose dictionaries because some types of code would quickly overload the dictionary with many negative entries. For name lookup the number of negative entries is bound by the number of global and builtin names referenced by the code that uses the dictionary as a namespace. This new type of slot has a surprisingly small impact on the rest of dictobject.c. Only one assertion had to be removed to accomodate it. All other code treats it as either an active entry (key !=NULL, !=dummy) or a deleted entry (value == NULL, key != NULL) and just happens to do the Right Thing for each case. If an entry with the same key as a negative entry is subsequently inserted into the dictionary it will overwrite the negative entry and be reflected immediately in the namespace. There is no caching and therefore no cache coherency issues. Known bugs: Negative entries do not resize the table. If there is not enough free space in the table they are simply not inserted. Assumes CACHE_HASH, INTERN_STRINGS without checking. Future directions: It should be possible to apply this to more than just LOAD_ATTR and LOAD_GLOBAL: attributes, modules, setting items, etc. The hit rate for the inline macro varies from 100% in most simple cases to 0% in cases where the first hash position in the table happens to be occupied by another entry. Even in these cases it is still very fast, but I want to get more consistent performance. I am starting to experiment with probabilistic techniques that shuffle entries in the hash table and try to ensure that the entries accessed most often are kept in the first hash positions as much as possible. Test code: * builtin.py class f: for i in xrange(10000000): hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; * global.py hex = 5 class f: for i in xrange(10000000): hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; * local.py class f: hex = 5 for i in xrange(10000000): hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; * fastlocal.py def f(): hex = 5 for i in xrange(10000000): hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; hex ; f() Oren

7 13

RE: [Python-Dev] Meta-reflections
by Moore, Paul Feb. 22, 2002

Feb. 22, 2002

From: Kevin Jacobs [mailto:jacobs@penguin.theopalgroup.com] > On Thu, 21 Feb 2002, Moore, Paul wrote: > > > I agree, but that "official" support has clear limitations. > > > > I'm not sure what you mean. > > When you request dir(object), there is a fairly significant > amount of work done. [...] > I know that we are talking about Python, and performance is > not of paramount importance. Hmm. I tend to favour "do it right, then do it fast". If there's a performance hit on dir(), why can't it be made faster? If nothing else, as a part of the core, dir() has the right to access __dict__ and __slots__. So there's no a priori reason why dir() should be slower than *any* user-coded way of doing the same. Of course, we *really* want vars() here, as we're otherwise doing work in dir() to get entries that we then throw away. But that's the only issue. Get vars() to work, and if it's too slow, you can argue that it's a bug because "I can get the same results by using the following code, which is faster". > I'm just not looking forward to 25% slowdowns in pickling > (number pulled out of hat) and I'm sure the Zope guys > aren't either... There's bound to be some slowdown, from the (new) need to find slots as well as dict-based attributes. I'm happy if you want it minimised. But that's a new point you've raised, which I don't have the expertise to comment on. > That is not the bug -- if for no other reason, the standard > library is free to use implementation specific knowledge. Getting > obj.__dict__ is a really slick and efficient way to reflect > on all normal instance variables. I'm not sure I agree here - it's better if the standard library uses interfaces which are available to the user. And if pickling can be made fast, why shouldn't the machinery that makes this possible be made available to the end user? You could say that this argues in favour of making __dict__ and __slots__ part of the "official" reflection API. My view is that it argues for making the "official" API (which I'm assuming will be vars() for now) efficient enough that people don't need to use __disct__ and __slots__. Encapsulation is good. > I don't see how filling slots with default values is > compatible with the premise that we want slots to act > as close to normal instance attributes as possible. Fair enough. I offered that as one option. Clearly you prefer the other (that what's in dir() and/or __slots__ cannot be guaranteed not to raise AttributeError). I'm happy either way, not having a vested interest in the issue. > > Why not just change the line stuff = object.__dict__ to > > > > stuff = [a for a in dir(object) if hasattr(object,a) and not > > callable(getattr(object,a))] > > Um, because its wrong? Sorry - it was an off-the-top-of-the-head suggestion. But it made my real point, which was that you can do it with dir(). > stuff = dict([ (a,getattr(object,a)) for a in vars(object) > if hasattr(object,a)]) > > Note that it does an unnecessary getattr, hasattr, memory > allocation and incurs loop overhead on every dict attribute, > but otherwise it should work once vars is fixed. Efficiency again. I'd have to bow to your greater experience here. Although with pickling, doesn't I/O usually outweigh any performance cost? > > 3. Document __dict__ as legacy usage, not slots-aware > > Agree, though __dict__ should still be a valid way of > accessing all non-slot instance attributes. Too much > legacy code would break if this were not so. That's what I meant. Document it as the historical way of getting at instance attributes. Still available, but code which uses it will not support slots. After all, if you pass classes using slots into code which uses __dict__, things will go wrong. That's just another sort of breakage. Nobody's arguing that __dict__ should go away. Except possibly from the documentation :-) > Calling base class setattr? I'm not sure what you mean? It's in the part of the descrintro document I pointed you at. Traditional implementations of setattr used assignment to self.__dict__['attr'] to avoid infinite recursion. The "new way" discussed in the descrintro document is to call the base class setattr. > By your logic, people don't have any business reading > __dict__, but they do. They don't have any business *any more*. An important distinction. (And it's not anywhere near as black and white as that comment implies - I know that). > Imagine what would happen if we didn't expose __dict__ in Python 2.3? Nothing at all, if we provide an alternative. Except for backward compatibility issues, which there's a well-documented deprecation process to address. Of course, nobody is proposing the removal of __dict__. All I'm suggesting is that we document its limitations, point out better ways, and leave it at that. Paul.

7 15