I was surprised to find in my profiling that instance variable access
was pretty slow.
I looked through the CPython code involved, and discovered something
that really surprises me.
Python, probably through the valid assumption that most attribute
lookups go to the class, tries to look for the attribute in the class
first, and in the instance, second.
What Python currently does is quite peculiar!
Here's a short description o PyObject_GenericGetAttr:
A. Python looks for a descriptor in the _entire_ mro hierarchy
(len(mro) class/type check and dict lookups).
B. If Python found a descriptor and it has both get and set functions
- it uses it to get the value and returns, skipping the next stage.
C. If Python either did not find a descriptor, or found one that has
no setter, it will try a lookup in the instance dict.
D. If Python failed to find it in the instance, it will use the
descriptor's getter, and if it has no getter it will use the
I believe the average costs of A are much higher than of C. Because
there is just 1 instance dict to look through, and it is also
typically smaller than the class dicts (in rare cases of worse-case
timings of hash lookups), while there are len(mro) dicts to look for a
This means that for simple instance variable lookups, Python is paying
the full mro lookup price!
I believe that this should be changed, so that Python first looks for
the attribute in the instance's dict and only then through the dict's
This will have the following effects:
A. It will break code that uses instance.__dict__['var'] directly,
when 'var' exists as a property with a __set__ in the class. I believe
this is not significant.
B. It will simplify getattr's semantics. Python should _always_ give
precedence to instance attributes over class ones, rather than have
very weird special-cases (such as a property with a __set__).
C. It will greatly speed up instance variable access, especially when
the class has a large mro.
I think obviously the code breakage is the worst problem. This could
probably be addressed by a transition version in which Python warns
about any instance attributes that existed in the mro as descriptors
What do you think?
> exists for a reason, wouldn't it be better if stdlib could cope with
> that at least with zipfile?
sys.setdefaultencoding just does not work. Many more things break when
you call it. It only exists because people like you insisted that it
> Also note that I'm trying to ask if zipfile should be improved, how it
> should be improved, and this possible improvement is not even for me
> (because now I know how zipfile behaves and I will work correctly with
> it, but someone else might stumble upon this very unexpectedly).
If you want to come up with a patch: sure. The zipfile module should
handle Unicode strings, encoding them in the encoding that the ZIP
specification defines (both the formal one, and the
The tricky question is what to do when reading in zipfiles with
non-ASCII characters (and yes, I understand that in your case
there were only ASCII characters in the file names).
> The problem was that sourcedir was unicode, and on my machine
> everything went ok multiple times. zipfile.ZipInfo.FileHeader would
> return unicode, but then when it writes it to a file it gets back to
> str (because mappings back and forth were identical). The problem
> happened when on a different machine header suddenly got byte 0x98 in
> position 10 (seems to be compress_size), which cp1251 codec couldn't
> decode. You see, arcname didn't even have unicode characters, but the
> mere fact that it was unicode made header upgrade to unicode in
> "return header + self.filename + self.extra".
Ok, now I understand. If filename is a Unicode string, header is
converted using the system encoding; depending on the exact value
of header and depending on the system encoding, this may cause
a decoding error.
This bug has been reported as http://bugs.python.org/1170311
> Because that's not supposed to work sanely when self.filename is
> unicode I'm asking if the right behavior would be to a) disallow
> unicode filenames in zipfile.ZipInfo, b) automatically convert
> filename to str in zipfile.ZipInfo, c) leave everything as it is.
The correct behavior would be b); the difficult details are what
encoding to use.
I was just looking through the code that handles frames (as part of my
current effort to determine how to improve on CPython's performance),
when I noticed the freelist/zombie mechanism for frame allocation
While the zombie mechanism seems like a nice optimization, I believe
there can be a couple of improvements.
Currently, each code object has a zombie frame that last executed it.
This zombie is reused when that code object is re-executed in a frame.
When a frame is released, it is reassigned as the zombie of the code,
and iff the code object already has a zombie assigned to it, it places
the frame in the freelist.
If I understand correctly, this means, that the "freelist" is actually
only ever used for recursive-call frames that were released. It also
means that these released frames will be reassigned to other code
objects in the future - in which case they will be reallocated,
perhaps unnecessarily. "freelist" is just temporary storage for
released recursive calls. A program with no recursions will always
have an empty freelist.
What is bounding the memory consumption of this mechanism is a limit
on the number of frames in the freelist (and the fact that there is a
limited number of code objects, each of which may have an additional
I believe a better way to implement this mechanism:
A. Replace the co_zombie frame with a co_zombie_freelist.
B. Remove the global freelist.
In other words, have a free list for each code object, rather than
one-per-code-object and a freelist.
This can be memory-bound by limiting the freelist size in each code object.
This can be use a bit more memory if a recursion is called just once -
and then discarded (waste for example 10 frames instead of 1), but can
save a lot of realloc calls when there is more than one recursion used
in the same program.
It is also possible to substantially increase the number of frames
stored per code-object, and then use some kind of more sophisticated
aging mechanism on the zombie freelists to periodically get rid of
unused freelists. That kind of mechanism would mean that even in the
case of recursive calls, virtually all frame allocs are available from
I also believe this to be somewhat simpler, as there is only one
concept (a zombie freelist) rather than 2 (a zombie code object and a
freelist for recursive calls), and frames are never realloc'd, but
Should I make a patch?
At 11:27 AM 6/10/2007 +0100, Gustavo Carneiro wrote:
> I have to agree with you. If removing support for
> self.__dict__['propertyname'] (where propertyname is also the name
> of a descriptor) is the price to pay for significant speedup, so be
> it. People doing that are asking for trouble anyway!
How so? This order of lookup is explicitly defined by the precedence
rules of PEP 252:
"""When a dynamic attribute (one defined in a regular object's
__dict__) has the same name as a static attribute (one defined
by a meta-object in the inheritance graph rooted at the regular
object's __class__), the static attribute has precedence if it
is a descriptor that defines a __set__ method (see below);
otherwise (if there is no __set__ method) the dynamic attribute
has precedence. In other words, for data attributes (those
with a __set__ method), the static definition overrides the
dynamic definition, but for other attributes, dynamic overrides
I fail to see how relying on explicitly-documented language behavior
is "asking for trouble".
At 04:14 AM 6/10/2007 +0300, Eyal Lotem wrote:
>On 6/10/07, Phillip J. Eby <pje(a)telecommunity.com> wrote:
> > At 12:23 AM 6/10/2007 +0300, Eyal Lotem wrote:
> > >A. It will break code that uses instance.__dict__['var'] directly,
> > >when 'var' exists as a property with a __set__ in the class. I believe
> > >this is not significant.
> > >B. It will simplify getattr's semantics. Python should _always_ give
> > >precedence to instance attributes over class ones, rather than have
> > >very weird special-cases (such as a property with a __set__).
> > Actually, these are features that are both used and desirable; I've
> > been using them both since Python 2.2 (i.e., for many years
> > now). I'm -1 on removing these features from any version of
> Python, even 3.0.
>It is the same feature, actually, two sides of the same coin.
>Why do you use self.__dict__['propertyname'] when you can use
Because I'm *not writing this by hand*. I'm using descriptors that
know what attribute name they're responsible for, and do the access directly.
>Why even call the first form, which is both longer and causes
>performance problems "a feature"?
If you don't understand that, IMO you don't yet understand enough
about the descriptor architecture to be proposing changes to it.
> > Note, by the way, that if you want to change attribute lookup
> > semantics, you can always override __getattribute__ and make it work
> > whatever way you like, without forcing everybody else to change
> *their* code.
>If I write my own __getattribute__ I lose the performance benefit that
>I am after.
Not if you write it in C.
>I do agree that code shouldn't be broken, that's why a transitional
>that requires using __fastlookup__ can be used (Unfortunately, from
>__future__ cannot be used as it is not local to a module, but to a
>class hierarchy - unless one imports a feature from __future__ into a
I have no idea what you're talking about here.
On 6/10/07, Phillip J. Eby <pje(a)telecommunity.com> wrote:
> At 12:23 AM 6/10/2007 +0300, Eyal Lotem wrote:
> >A. It will break code that uses instance.__dict__['var'] directly,
> >when 'var' exists as a property with a __set__ in the class. I believe
> >this is not significant.
> >B. It will simplify getattr's semantics. Python should _always_ give
> >precedence to instance attributes over class ones, rather than have
> >very weird special-cases (such as a property with a __set__).
> Actually, these are features that are both used and desirable; I've
> been using them both since Python 2.2 (i.e., for many years
> now). I'm -1 on removing these features from any version of Python, even 3.0.
It is the same feature, actually, two sides of the same coin.
Why do you use self.__dict__['propertyname'] when you can use
Why even call the first form, which is both longer and causes
performance problems "a feature"?
> >C. It will greatly speed up instance variable access, especially when
> >the class has a large mro.
> ...at the cost of slowing down access to properties and __slots__, by
> adding an *extra* dictionary lookup there.
It will slow down access to properties - but that slowdown is insignificant:
A. The vast majority of lookups are *NOT* of properties. They are the
rare case and should not be the focus of optimization.
B. Property access involves calling Python functions - which is
heavier than a single dict lookup.
C. The dict lookup to find the property in the __mro__ can involve
many dicts (so in those cases adding a single dict lookup is not
> Note, by the way, that if you want to change attribute lookup
> semantics, you can always override __getattribute__ and make it work
> whatever way you like, without forcing everybody else to change *their* code.
If I write my own __getattribute__ I lose the performance benefit that
I am after.
I do agree that code shouldn't be broken, that's why a transitional
that requires using __fastlookup__ can be used (Unfortunately, from
__future__ cannot be used as it is not local to a module, but to a
class hierarchy - unless one imports a feature from __future__ into a
Today I've stumbled upon a bug in my program that wasn't very
straightforward to understand. The problem is that I was passing
unicode filenames to zipfile.ZipFile.write and I had
sys.setdefaultencoding() in effect, which resulted in a situation
where most of the bytes generated in zipfile.ZipInfo.FileHeader would
pass thru, except for a few, which caused codec error on another
machine (where filenames got infectiously upgraded to unicode). The
problem here is that it was absolutely unclear at first that I get
unicode filenames passed to write, and it incorrectly accepted them
silently. Is it worth to submit a bug report on this? The desired
behavior here would be to either a) disallow unicode strings as
arcname are raise an exception (since it is used in concatenation with
raw data it is likely to cause problems because of auto upgrading raw
data to unicode), or b) silently encode unicode strings to raw strings
(something like if isinstance(filename, unicode): filename =
filename.encode() in zipfile.ZipInfo constructor).
So, should I submit a bug report, and which behavior would be actually correct?
ACTIVITY SUMMARY (06/03/07 - 06/10/07)
Tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue
number. Do NOT respond to this message.
1645 open ( +0) / 8584 closed ( +0) / 10229 total ( +0)
Average duration of open issues: 822 days.
Median duration of open issues: 770 days.
Open Issues Breakdown
open 1645 ( +0)
pending 0 ( +0)
Issues Now Closed (1)
New issue test for email 87 days
> On 6/10/07, Steven Bethard <steven.bethard(a)gmail.com> wrote:
> > On 6/9/07, Eyal Lotem <eyal.lotem(a)gmail.com> wrote:
> > > I believe that this should be changed, so that Python first looks for
> > > the attribute in the instance's dict and only then through the dict's
> > > mro.
> > Are you suggesting that the following code should print "43" instead of "42"?
> > ::
> > >>> class C(object):
> > ... x = property(lambda self: self._x)
> > ... def __init__(self):
> > ... self._x = 42
> > ...
> > >>> c = C()
> > >>> c.__dict__['x'] = 43
> > >>> c.x
> > 42
On 6/9/07, Eyal Lotem <eyal.lotem(a)gmail.com> wrote:
> Yes, I do suggest that.
> But its important to notice that this is not a suggestion in order to
> improve Python, but one that makes it possible to get reasonable
> performance out of CPython. As such, I don't believe it should be done
> in Py3K.
> Firstly, like everything that breaks backwards compatibility, it is
> possible to have a transitional version that spits warnings for all
> problems (detect name clashes between properties and instance dict).
Sure, but then you're talking about really introducing this in Python
2.7, with 2.6 as a transitional version. So take a minute to look at
the release timelines:
The initial 2.6 target is for April 2008.
I hope to have a first alpha release (3.0a1) out in the first half of
2007; it should take no more than a year from then before the first
proper release, named Python 3.0
So I'm expecting Python 3.0 to come out *before* 2.7. Thus if you're
proposing a backwards-incompatible change that would have to wait
until 2.7 anyway, why not propose it for 3.0 where
backwards-incompatible changes are more acceptable?
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
--- Bucky Katt, Get Fuzzy
I am having a couple of issues compiling Python 2.5.1 under Sun Solaris
Studio 11 on Solaris 8.
Everything compiles correctly except the _ctypes module because it
cannot use the libffi that comes with Python and it does not exist on
Has anyone gotten it to compile correctly using Studio 11? I know it
will compile under GCC but I am not allowed to use GCC.
Also, during the pyexpat tests, Python generates a segfault.
Are there any patches to fix these?