Mailman 3 February 2002 - Python-Dev

Python 2.2 group missing on SF Patches
by Neal Norwitz Feb. 11, 2002

Feb. 11, 2002

There is no 2.2 (or 2.2.1) choice under Group when submitting a patch on Source Forge. Neal

2 1

Re: [Python-Dev] Want to co-design and implement a logging module?
by Michael Hudson Feb. 11, 2002

Feb. 11, 2002

Greg Ward <gward(a)python.net> writes: > On 04 February 2002, Trent Mick volunteered to "do something" > about a standard logging module for Python: > > How about I try to have a PEP together within a week or two, and perhaps a > > working base implementation? > > Well, it's been exactly 7 days. Trent, are you halfway done yet? ;-) > > (Yes, I've been thinking that the solution to the Distutils' verbosity > problem lies somewhere down this road.) But I believe that 1.5.2 compatibility is still relavent for distutils, so a logging module in 2.3 is not especially helpful, unless one can come up with some scheme whereby the standalone distutils packages can use a bundled logger and the 2.3 distutils use the library one. I had a go at implementing a very KISS approach to distutils logging this morning and found what I was doing conflicted horribly with distutils' current practice, so I stopped. Cheers, M. -- Also, remember to put the galaxy back when you've finished, or an angry mob of astronomers will come round and kneecap you with a small telescope for littering. -- Simon Tatham, ucam.chat, from Owen Dunn's review of the year

2 1

Speeding up instance attribute access
by Guido van Rossum Feb. 11, 2002

Feb. 11, 2002

(By mistake I sent an incomplete version of this earlier. Please ignore that and read this instead.) Inspired by the second half of Jeremy's talk on DevDay, here's my alternative approach for speeding up instance attribute access. Like my idea for globals, it uses double indirection rather than recompilation. - We only care about attributes of 'self' (which is identified as the first argument of a method, not by name). We can exclude functions from our analysis that make any assignment to self -- this is extremely rare and would throw off our analysis. We should also exclude static methods and class methods, since their first argument doesn't have the same role. - Static analysis of the source code of a class (without access to the base class) can determine attributes of the class, and to some extent instance variables. Without also analyzing the base classes, this analysis cannot reliably distinguish between instance variables and methods inherited from a base class; it can distinguish between instance variables and methods defined in the current class. - We can guess the status of un-assigned-to inherited attributes by seeing whether they are called or not. This is not 100% accurate, so we need things to work (if slower) even when we guess wrong. - For instance variable references and stores of the form self.<name>, the bytecode compiler emits opcodes LOAD_SELF_IVAR and STORE_SELF_IVAR , where is a small int identifying the instance variable (ivar). A particular ivar is identified by the same throughout all methods defined in the same class statement, but there is no attempt to coordinate this across different classes related by inheritance. - Some data structure describing the mapping from to attribute name, and whether it's an ivar or a method, is produced by the compiler and stored in the class, in a way that a user can't change it. The function objects representing methods also contain a pointer to this data structure. (Or the code objects? But it needs to be shared. Details, details.) - At *run time*, when a class object is created, another data structure is created that accumulates the -to-name mappings from that class and all its base classes. This has more accurate information because it can collect information from the bases (though it can still be fooled by dynamic manipulations of classes). In particular, it has the canonical set of known instance variables for instances of this class. This is stored in the class object, in a way that a user can't change it. - When an instance is created, the run-time data structure stored in the class is consulted to know how many instance variables to allocate. An array is allocated at the end of the instance (or in a separately allocated block?) with one PyObject pointer for each known instance variable. There's also a pointer to the instance __dict__, to hold instance variables that the parser didn't spot, but this pointer starts off as NULL -- a dictionary is created for it only when needed. - There is no requirement that the layout of the ivar array for instances of a subclass is an extension of the layout of the ivar array for its base classes. But there *is* a requirement that all instances of the same class that are not instances of a subclass (i.e., all x and y where x.__class__ is y.__class__) have the same layout, and this layout is determined by the run-time data structure stored in the class. - Now all we need is an efficient way to map LOAD_SELF_IVAR to an index in the array of ivars. Two different classes play a role here: the class of self (the run-time class) and the class that defined the method containing the LOAD_SELF_IVAR opcode (the compile-time class). We assume the run-time class is a subclass of the compile-time class. The correct mapping can easily be calculated from the data structures left behind by the compiler in the compile-time class and by class construction at run time in the run-time class. Since this mapping doesn't change, it can be calculated once and then cached. The cache can be held in the run-time class; it can be a dictionary whose keys are compile-time classes. This means a single dict lookup that must be done once per method call (and only when LOAD_SELF_IVAR or STORE_SELF_IVAR is used). We could save even that dict lookup in most cases by caching the run-time class and the outcome with the method. - We need fallbacks for various exceptional cases: * If the compile-time class uses LOAD_SELF_IVAR but the run-time class doesn't think that is an instance variable, LOAD_SELF_IVAR must fall back to look in the instance dict (if non-NULL) and then down the run-time class and its base classes. * If the ivar slot in the instance corresponding to exists but is NULL, LOAD_SELF_IVAR must fall back to searching the run-time class and its base classes. For both cases, the mapping from to attribute names must be available. The language and the current code generation guarantee that 'self' is the first local variable. - The instance __dict__ must become a proxy that knows whether a given name is stored in the array of ivars or in the overflow dict; this is much like Jeremy's DLict. - Note that there are two sources of savings: the major savings (probably) comes from avoiding a dict lookup for every ivar access; an additional minor savings comes from collapsing two opcodes: LOAD_FAST 0 (self) LOAD_ATTR 1 (foo) into one: LOAD_SELF_IVAR 0 (self.foo) - I don't know if we should try to generate LOAD_SELF_IVAR only for things that really are (likely) ivars, or for all attributes. Maybe it would be nice if we also had a single-opcode way to express a method call on self, e.g. CALL_SELF_METHOD , <n>, <k> where identifies the method, and <n> and <k> are the number of positional and keyword arguments. Or maybe we should just have LOAD_SELF_METHOD which looks in the instance overflow dict (but only if non-NULL) and in the class and bases, but can avoid looking in the ivars array if does not describe a known ivar (again this information can be cached). - The required global analysis is a bit hairy, and not something we already do. I believe that Jeremy thinks PyChecker already does this; I'm not sure if that means we can borrow code or just ideas. --Guido van Rossum (home page: http://www.python.org/~guido/)

3 3

Re: [Zope3-dev] Strange behaviour of python2.2 -U with Zope 3
by Andreas Jung Feb. 10, 2002

Feb. 10, 2002

Another followup: the module import seems to completely broken when using the -U option. Andreas ----- Original Message ----- From: "Andreas Jung" <andreas(a)andreas-jung.com> To: <zope3-dev(a)zope.org> Sent: Friday, February 08, 2002 19:48 Subject: [Zope3-dev] Strange behaviour of python2.2 -U with Zope 3 > python2.2 utilities/unittestgui.py Zope.Testing.allZopeTests > start the GUI for the Zope3 unittests. So far so good. > > I tried to run all tests with unicode as default string type: > > python2.2 -U utilities/unittestgui.py Zope.Testing.allZopeTests > > This fails with the following traceback: > > yetix@/develop/DC/sandboxes/3x(57)% python2.2 -U utilities/unittestgui.py > Zope.Testing.allZopeTests > Traceback (most recent call last): > File "utilities/unittestgui.py", line 30, in ? > import linecache > ImportError: No module linecache > > Also "python2.2 -U -c "import linecache" " fails > > Any ideas ? > > Andreas > > > > _______________________________________________ > Zope3-dev mailing list > Zope3-dev(a)zope.org > http://lists.zope.org/mailman/listinfo/zope3-dev >

2 1

You know your email is vulnerable to SPAM Robots?
by Jeff Graysmith Feb. 9, 2002

Feb. 9, 2002

Hello, Please pardon the intrusion, but I saw that the email address python- dev(a)python.org is in plain text on your site at http://python.sourceforge.net/peps/pep-0226.html making it vulnerable to be harvested by SPAM robots. Check this out there's a way to hide your email from robots, but still have it visible to human users. http://www.email-cloak.net Sincerely, Jeff Graysmith

1 0

Python optmizations
by Gustavo Niemeyer Feb. 8, 2002

Feb. 8, 2002

Hello Skip! I've been reading some books and papers about stack virtual machines optimization, and playing around with Python's bytecode and inner loop organization. As always, I found some interesting results and some frustrating ones. Recently, I have found your paper about peephole optimization, and other tries you've made in the same job. Well, basically I discovered that I'm not original, and repeated most of your ideas and mistakes. :-) But that's ok. It gave me a good idea of paths to follow if I want to keep playing with this. One thing I thought and also found a reference in your paper is about some instructions that should be turned into a single opcode. To understand how this would affect the code, I have disassembled the whole Python standard library, and the whole Zope library. After that I've run a script to detect opcode repeatings (excluding SET_LINENO). Here are the top repeatings: 23632 LOAD_FAST, LOAD_ATTR 15382 LOAD_CONST, LOAD_CONST 12842 JUMP_IF_FALSE, POP_TOP 12397 CALL_FUNCTION, POP_TOP 12121 LOAD_FAST, LOAD_FAST Not by casuality, I found in your paper references to a LOAD_FAST_ATTR opcode. Since you probably have mentioned this to others, I wouldn't like to bother everyone again asking why it was not implemented. Could you please explain me the reasons that left this behind? If you have the time, I'd also like to understand what's the trouble involved in getting a peephole optimizer in the python compiler itself. Is it just about compiling performance? I don't remember to have read about this in your paper, but you probably thought about that as well. Thank you! -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ]

2 2

Extending types in C - help needed
by Jack Jansen Feb. 8, 2002

Feb. 8, 2002

In the discussion on my request for an ("O@", typeobject, void **) format for PyArg_Parse and Py_BuildValue MAL suggested that I could get the same functionality by creating a type WrapperTypeObject, which would be a subtype of TypeObject with extra fields pointing to the _New() and _Convert() routines to convert Python objects from/to C pointers. This would be good enough for me, because then types wanting to participate in the wrapper protocol would subtype WrapperTypeObject in stead of TypeObject, and two global routines could return the _New and _Convert routines given the type object, and we wouldn't need yet another PyArg_Parse format specifier. However, after digging high and low I haven't been able to deduce how I would then use this WrapperType in C as the type for my extension module objects. Are there any examples? If not, could someone who understands the new inheritance scheme give me some clues as to how to do this? -- - Jack Jansen <Jack.Jansen(a)oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

7 30

Speeding up instance attribute access
by Guido van Rossum Feb. 8, 2002

Feb. 8, 2002

Inspired by the second half of Jeremy's talk on DevDay, here's my alternative approach for speeding up instance attribute access. Like my idea for globals, it uses double indirection rather than recompilation. - We only care about attributes of 'self' (which is identified as the first argument of a method, not by name). We can exclude functions from our analysis that make any assignment to self -- this is extremely rare and would throw off our analysis. We should also exclude static methods and class methods, since their first argument doesn't have the same role. - Static analysis of the source code of a class (without access to the base class) can determine attributes of the class, and to some extent instance variables. Without also analyzing the base classes, this analysis cannot reliably distinguish between instance variables and methods inherited from a base class; it can distinguish between instance variables and methods defined in the current class. - We can guess the status of un-assigned-to inherited attributes by seeing whether they are called or not. This is not 100% accurate, so we need things to work (if slower) even when we guess wrong. - For instance variable references and stores of the form self.<name>, the bytecode compiler emits opcodes LOAD_SELF_IVAR and STORE_SELF_IVAR , where is a small int identifying the instance variable (ivar). A particular ivar is identified by the same throughout all methods defined in the same class statement, but there is no attempt to coordinate this across different classes related by inheritance. - It would be nice if we also had a single-opcode way to express a method call on self, e.g. CALL_SELF_METHOD , <n>, <k> where identifies the method like above, and <n> and <k> are the number of positional and keyword arguments. Or maybe we should just have LOAD_SELF_METHOD which may be able to skip looking in the instance dict. - Some data structure describing the mapping from to attribute name, and whether it's an ivar or a method, is produced by the compiler and stored in the class __dict__. The function objects representing methods also contain a pointer to this data structure. (Or the code objects? But it needs to be shared. Details, details.) - When a class object is created (at run-time), another data structure is created that accumulates the -to-name mappings from that class and all its base classes.

1 1

Re: [Python-Dev] Mixing memory management APIs
by Michael Hudson Feb. 7, 2002

Feb. 7, 2002

"M.-A. Lemburg" <mal(a)lemburg.com> writes: > Neal Norwitz wrote: > > > "M.-A. Lemburg" wrote: > > > > > >>I've checked in a patch for the UTF-8 codec problem. Could you > >>try Purify against the CVS version ? > >> > > > > with-pymalloc or without or both? > > > Both if possible -- the leakage showed up with pymalloc AFAIR :-) I thought we were chasing memory stomping, not leaking, this time around... Cheers, M. -- /* I'd just like to take this moment to point out that C has all the expressive power of two dixie cups and a string. */ -- Jamie Zawinski from the xkeycaps source

2 1

Re: [Python-Dev] Mixing memory management APIs
by Michael Hudson Feb. 7, 2002

Feb. 7, 2002

Neal Norwitz <neal(a)metaslash.com> writes: > Because of Michael Hudson's request, I tried running Purify > --with-pymalloc enabled. The results were a bit surprising: 13664 errors! > > All the errors were in unicodeobject.c. There were 3 types of errors: > Free Memory Reads, Array Bounds Reads, and Unitialized Memory Reads. > The line #s were in strange places (e.g., in a function declaration > and accessing self->length in an if clause, after it was accessed w/o error). > The line #s are primarily: unicodeobject.c:2875, and unicodeobject.c:2214. Might this have something to do with bug [ #495401 ] Build troubles: --with-pymalloc http://sourceforge.net/tracker/?func=detail&atid=105470&aid=495401&group_id… ? Is there a reason one of the fixes for this problem hasn't been checked in yet? Cheers, M. -- . <- the point your article -> . |------------------------- a long way ------------------------| -- Cristophe Rhodes, ucam.chat

3 9