Mailman 3 June 2007 - Python-Dev

Santa Fe Python Day report
by Facundo Batista June 11, 2007

June 11, 2007

It was very succesful, around +300 people assisted, and there were a lot of interesting talks (two introductory talks, Turbogears, PyWeek, Zope 3, security, creating 3D games, Plone, automatic security testings, concurrency, and programming the OLPC). I want to thanks the PSF for the received support. Python is developing interestingly in Argentina, and this Python Days are both a prove of that, and a way to get more Python developers. Some links: Santa Fe Python Day: http://www.python-santafe.com.ar/ Python Argentina: http://www.python.com.ar/moin Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

1 0

Instance variable access and descriptors
by Eyal Lotem June 11, 2007

June 11, 2007

Hi. I was surprised to find in my profiling that instance variable access was pretty slow. I looked through the CPython code involved, and discovered something that really surprises me. Python, probably through the valid assumption that most attribute lookups go to the class, tries to look for the attribute in the class first, and in the instance, second. What Python currently does is quite peculiar! Here's a short description o PyObject_GenericGetAttr: A. Python looks for a descriptor in the _entire_ mro hierarchy (len(mro) class/type check and dict lookups). B. If Python found a descriptor and it has both get and set functions - it uses it to get the value and returns, skipping the next stage. C. If Python either did not find a descriptor, or found one that has no setter, it will try a lookup in the instance dict. D. If Python failed to find it in the instance, it will use the descriptor's getter, and if it has no getter it will use the descriptor itself. I believe the average costs of A are much higher than of C. Because there is just 1 instance dict to look through, and it is also typically smaller than the class dicts (in rare cases of worse-case timings of hash lookups), while there are len(mro) dicts to look for a descriptor in. This means that for simple instance variable lookups, Python is paying the full mro lookup price! I believe that this should be changed, so that Python first looks for the attribute in the instance's dict and only then through the dict's mro. This will have the following effects: A. It will break code that uses instance.__dict__['var'] directly, when 'var' exists as a property with a __set__ in the class. I believe this is not significant. B. It will simplify getattr's semantics. Python should _always_ give precedence to instance attributes over class ones, rather than have very weird special-cases (such as a property with a __set__). C. It will greatly speed up instance variable access, especially when the class has a large mro. I think obviously the code breakage is the worst problem. This could probably be addressed by a transition version in which Python warns about any instance attributes that existed in the mro as descriptors as well. What do you think?

4 3

Re: [Python-Dev] zipfile and unicode filenames
by "Martin v. Löwis" June 11, 2007

June 11, 2007

> sys.setdefaultencoding() > exists for a reason, wouldn't it be better if stdlib could cope with > that at least with zipfile? sys.setdefaultencoding just does not work. Many more things break when you call it. It only exists because people like you insisted that it exists. > Also note that I'm trying to ask if zipfile should be improved, how it > should be improved, and this possible improvement is not even for me > (because now I know how zipfile behaves and I will work correctly with > it, but someone else might stumble upon this very unexpectedly). If you want to come up with a patch: sure. The zipfile module should handle Unicode strings, encoding them in the encoding that the ZIP specification defines (both the formal one, and the informal-defined-by-pkwares-implementation). The tricky question is what to do when reading in zipfiles with non-ASCII characters (and yes, I understand that in your case there were only ASCII characters in the file names). > The problem was that sourcedir was unicode, and on my machine > everything went ok multiple times. zipfile.ZipInfo.FileHeader would > return unicode, but then when it writes it to a file it gets back to > str (because mappings back and forth were identical). The problem > happened when on a different machine header suddenly got byte 0x98 in > position 10 (seems to be compress_size), which cp1251 codec couldn't > decode. You see, arcname didn't even have unicode characters, but the > mere fact that it was unicode made header upgrade to unicode in > "return header + self.filename + self.extra". Ok, now I understand. If filename is a Unicode string, header is converted using the system encoding; depending on the exact value of header and depending on the system encoding, this may cause a decoding error. This bug has been reported as http://bugs.python.org/1170311 > Because that's not supposed to work sanely when self.filename is > unicode I'm asking if the right behavior would be to a) disallow > unicode filenames in zipfile.ZipInfo, b) automatically convert > filename to str in zipfile.ZipInfo, c) leave everything as it is. The correct behavior would be b); the difficult details are what encoding to use. Regards, Martin

2 9

Frame zombies
by Eyal Lotem June 11, 2007

June 11, 2007

I was just looking through the code that handles frames (as part of my current effort to determine how to improve on CPython's performance), when I noticed the freelist/zombie mechanism for frame allocation handling. While the zombie mechanism seems like a nice optimization, I believe there can be a couple of improvements. Currently, each code object has a zombie frame that last executed it. This zombie is reused when that code object is re-executed in a frame. When a frame is released, it is reassigned as the zombie of the code, and iff the code object already has a zombie assigned to it, it places the frame in the freelist. If I understand correctly, this means, that the "freelist" is actually only ever used for recursive-call frames that were released. It also means that these released frames will be reassigned to other code objects in the future - in which case they will be reallocated, perhaps unnecessarily. "freelist" is just temporary storage for released recursive calls. A program with no recursions will always have an empty freelist. What is bounding the memory consumption of this mechanism is a limit on the number of frames in the freelist (and the fact that there is a limited number of code objects, each of which may have an additional zombie frame). I believe a better way to implement this mechanism: A. Replace the co_zombie frame with a co_zombie_freelist. B. Remove the global freelist. In other words, have a free list for each code object, rather than one-per-code-object and a freelist. This can be memory-bound by limiting the freelist size in each code object. This can be use a bit more memory if a recursion is called just once - and then discarded (waste for example 10 frames instead of 1), but can save a lot of realloc calls when there is more than one recursion used in the same program. It is also possible to substantially increase the number of frames stored per code-object, and then use some kind of more sophisticated aging mechanism on the zombie freelists to periodically get rid of unused freelists. That kind of mechanism would mean that even in the case of recursive calls, virtually all frame allocs are available from the freelist. I also believe this to be somewhat simpler, as there is only one concept (a zombie freelist) rather than 2 (a zombie code object and a freelist for recursive calls), and frames are never realloc'd, but only allocated. Should I make a patch?

2 5

Re: [Python-Dev] Fwd: Instance variable access and descriptors
by Phillip J. Eby June 10, 2007

June 10, 2007

At 11:27 AM 6/10/2007 +0100, Gustavo Carneiro wrote: > I have to agree with you. If removing support for > self.__dict__['propertyname'] (where propertyname is also the name > of a descriptor) is the price to pay for significant speedup, so be > it. People doing that are asking for trouble anyway! How so? This order of lookup is explicitly defined by the precedence rules of PEP 252: """When a dynamic attribute (one defined in a regular object's __dict__) has the same name as a static attribute (one defined by a meta-object in the inheritance graph rooted at the regular object's __class__), the static attribute has precedence if it is a descriptor that defines a __set__ method (see below); otherwise (if there is no __set__ method) the dynamic attribute has precedence. In other words, for data attributes (those with a __set__ method), the static definition overrides the dynamic definition, but for other attributes, dynamic overrides static.""" I fail to see how relying on explicitly-documented language behavior is "asking for trouble".

1 0

Re: [Python-Dev] Fwd: Instance variable access and descriptors
by Phillip J. Eby June 10, 2007

June 10, 2007

At 04:14 AM 6/10/2007 +0300, Eyal Lotem wrote: >On 6/10/07, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > At 12:23 AM 6/10/2007 +0300, Eyal Lotem wrote: > > >A. It will break code that uses instance.__dict__['var'] directly, > > >when 'var' exists as a property with a __set__ in the class. I believe > > >this is not significant. > > >B. It will simplify getattr's semantics. Python should _always_ give > > >precedence to instance attributes over class ones, rather than have > > >very weird special-cases (such as a property with a __set__). > > > > Actually, these are features that are both used and desirable; I've > > been using them both since Python 2.2 (i.e., for many years > > now). I'm -1 on removing these features from any version of > Python, even 3.0. > >It is the same feature, actually, two sides of the same coin. >Why do you use self.__dict__['propertyname'] when you can use >self._propertyname? Because I'm *not writing this by hand*. I'm using descriptors that know what attribute name they're responsible for, and do the access directly. >Why even call the first form, which is both longer and causes >performance problems "a feature"? If you don't understand that, IMO you don't yet understand enough about the descriptor architecture to be proposing changes to it. > > Note, by the way, that if you want to change attribute lookup > > semantics, you can always override __getattribute__ and make it work > > whatever way you like, without forcing everybody else to change > *their* code. >If I write my own __getattribute__ I lose the performance benefit that >I am after. Not if you write it in C. >I do agree that code shouldn't be broken, that's why a transitional >that requires using __fastlookup__ can be used (Unfortunately, from >__future__ cannot be used as it is not local to a module, but to a >class hierarchy - unless one imports a feature from __future__ into a >class). I have no idea what you're talking about here.

1 0

Fwd: Instance variable access and descriptors
by Eyal Lotem June 10, 2007

June 10, 2007

On 6/10/07, Phillip J. Eby <pje(a)telecommunity.com> wrote: > At 12:23 AM 6/10/2007 +0300, Eyal Lotem wrote: > >A. It will break code that uses instance.__dict__['var'] directly, > >when 'var' exists as a property with a __set__ in the class. I believe > >this is not significant. > >B. It will simplify getattr's semantics. Python should _always_ give > >precedence to instance attributes over class ones, rather than have > >very weird special-cases (such as a property with a __set__). > > Actually, these are features that are both used and desirable; I've > been using them both since Python 2.2 (i.e., for many years > now). I'm -1 on removing these features from any version of Python, even 3.0. It is the same feature, actually, two sides of the same coin. Why do you use self.__dict__['propertyname'] when you can use self._propertyname? Why even call the first form, which is both longer and causes performance problems "a feature"? > >C. It will greatly speed up instance variable access, especially when > >the class has a large mro. > > ...at the cost of slowing down access to properties and __slots__, by > adding an *extra* dictionary lookup there. It will slow down access to properties - but that slowdown is insignificant: A. The vast majority of lookups are *NOT* of properties. They are the rare case and should not be the focus of optimization. B. Property access involves calling Python functions - which is heavier than a single dict lookup. C. The dict lookup to find the property in the __mro__ can involve many dicts (so in those cases adding a single dict lookup is not heavy). > Note, by the way, that if you want to change attribute lookup > semantics, you can always override __getattribute__ and make it work > whatever way you like, without forcing everybody else to change *their* code. If I write my own __getattribute__ I lose the performance benefit that I am after. I do agree that code shouldn't be broken, that's why a transitional that requires using __fastlookup__ can be used (Unfortunately, from __future__ cannot be used as it is not local to a module, but to a class hierarchy - unless one imports a feature from __future__ into a class).

2 1

zipfile and unicode filenames
by Alexey Borzenkov June 10, 2007

June 10, 2007

Hi everyone, Today I've stumbled upon a bug in my program that wasn't very straightforward to understand. The problem is that I was passing unicode filenames to zipfile.ZipFile.write and I had sys.setdefaultencoding() in effect, which resulted in a situation where most of the bytes generated in zipfile.ZipInfo.FileHeader would pass thru, except for a few, which caused codec error on another machine (where filenames got infectiously upgraded to unicode). The problem here is that it was absolutely unclear at first that I get unicode filenames passed to write, and it incorrectly accepted them silently. Is it worth to submit a bug report on this? The desired behavior here would be to either a) disallow unicode strings as arcname are raise an exception (since it is used in concatenation with raw data it is likely to cause problems because of auto upgrading raw data to unicode), or b) silently encode unicode strings to raw strings (something like if isinstance(filename, unicode): filename = filename.encode() in zipfile.ZipInfo constructor). So, should I submit a bug report, and which behavior would be actually correct?

2 1

Summary of Tracker Issues
by Tracker June 10, 2007

June 10, 2007

ACTIVITY SUMMARY (06/03/07 - 06/10/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1645 open ( +0) / 8584 closed ( +0) / 10229 total ( +0) Average duration of open issues: 822 days. Median duration of open issues: 770 days. Open Issues Breakdown open 1645 ( +0) pending 0 ( +0) Issues Now Closed (1) _____________________ New issue test for email 87 days http://bugs.python.org/issue1001 admin

1 0

Re: [Python-Dev] Instance variable access and descriptors
by Steven Bethard June 9, 2007

June 9, 2007

> On 6/10/07, Steven Bethard <steven.bethard(a)gmail.com> wrote: > > On 6/9/07, Eyal Lotem <eyal.lotem(a)gmail.com> wrote: > > > I believe that this should be changed, so that Python first looks for > > > the attribute in the instance's dict and only then through the dict's > > > mro. > > > > Are you suggesting that the following code should print "43" instead of "42"? > > :: > > > > >>> class C(object): > > ... x = property(lambda self: self._x) > > ... def __init__(self): > > ... self._x = 42 > > ... > > >>> c = C() > > >>> c.__dict__['x'] = 43 > > >>> c.x > > 42 On 6/9/07, Eyal Lotem <eyal.lotem(a)gmail.com> wrote: > Yes, I do suggest that. > But its important to notice that this is not a suggestion in order to > improve Python, but one that makes it possible to get reasonable > performance out of CPython. As such, I don't believe it should be done > in Py3K. > > Firstly, like everything that breaks backwards compatibility, it is > possible to have a transitional version that spits warnings for all > problems (detect name clashes between properties and instance dict). Sure, but then you're talking about really introducing this in Python 2.7, with 2.6 as a transitional version. So take a minute to look at the release timelines: http://www.python.org/dev/peps/pep-0361/ The initial 2.6 target is for April 2008. http://www.python.org/dev/peps/pep-3000/ I hope to have a first alpha release (3.0a1) out in the first half of 2007; it should take no more than a year from then before the first proper release, named Python 3.0 So I'm expecting Python 3.0 to come out *before* 2.7. Thus if you're proposing a backwards-incompatible change that would have to wait until 2.7 anyway, why not propose it for 3.0 where backwards-incompatible changes are more acceptable? STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

1 0