__slots_ and inheritance

[Alex Martelli]

A. Lloyd Flanagan wrote:

> Alexander Schmolck <a.schmolck at gmx.net> wrote in message
> news:<yfsistmh4ec.fsf at black132.ex.ac.uk>...
>  
>> __slots__ is the one addition in python2.2 that really annoys me. It
>> seems an ill-conceived jumble of conceptually unrelated things (bug
>> avoidance, optimization, safety, encapsulation) that screws up reflection
>> big time for little apparent gain. As Knuth said ...
>> 
>> 'as
> 
> I agree.  The more I looked at it, the more it became clear that
> __slots__ turns a class into something very non-pythonic, with
> significantly different behavior.  A lot of utilities, debuggers, and
> advanced programming tricks will be coming with this disclaimer:
> "does not work with any class which defines __slots__"

I think it's an important memory optimization, PERIOD.  The other
conceptually unrelated things (bug avoidance -- I can't even SEE the
others!) aren't in __slots__' own design (even though a commentator
who's usually quite perceptive seems to have seen the "bug avoidance"
feature that wasn't really there).

As Knuth said, *premature* optimization is the root of all evil in
programming.  But __slots__ may allow very substantial optimization
that need not be premature.  I think it should just have been more
clearly documented as a pure optimization issue right from the start,
that's all.


> Unless you absolutely have to have it, don't use it.

"Absolutely have to have it", just like "screws up big time", are
silly overbids IMHO -- over-reactions that I don't see as justified
at all!  Why can't we all just keep SOME perspective, please...?!

Consider a typical use case, a class whose instances have and need
a single attribute 'x', and of which we need to generate 10,000:

[alex at lancelot python2.3]$ python -O timeit.py -s 'class X(object):' -s' def 
__init__(self): self.x=23' 'z=[X() for i in xrange(9999)]'
10 loops, best of 3: 6.51e+04 usec per loop
[alex at lancelot python2.3]$ python -O timeit.py -s 'class X(object):' -s' def 
__init__(self): self.x=23' 'z=[X() for i in xrange(9999)]'
10 loops, best of 3: 6.56e+04 usec per loop
[alex at lancelot python2.3]$ python -O timeit.py -s 'class X(object):' -s ' 
__slots__=["x"]' -s' def __init__(self): self.x=23' 'z=[X() for i in 
xrange(9999)]'
10 loops, best of 3: 3.36e+04 usec per loop
[alex at lancelot python2.3]$ python -O timeit.py -s 'class X(object):' -s ' 
__slots__=["x"]' -s' def __init__(self): self.x=23' 'z=[X() for i in 
xrange(9999)]'
10 loops, best of 3: 3.34e+04 usec per loop
[alex at lancelot python2.3]$

Do we "*absolutely* have to have" the roughly factor-of-2 speedup given
by __slots__?  Clearly not -- we COULD live with a program that's
roughly twice as slow, and here we're measuring only initialization
anyway.  But, consider also the memory-churning, particularly when
MORE instances of class X need to be around, say 100,000...:

[alex at lancelot python2.3]$ time python -O timeit.py -s 'class X(object):' -s 
' __slots__=["x"]' -s' def __init__(self): self.x=23' 'z=[X() for i in 
xrange(99999)]'
10 loops, best of 3: 4.28e+05 usec per loop
16.49user 0.74system 0:17.58elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (507major+5528minor)pagefaults 0swaps

[alex at lancelot python2.3]$ time python -O timeit.py -s 'class X(object):' 
-s' def __init__(self): self.x=23' 'z=[X() for i in xrange(99999)]'
10 loops, best of 3: 1.19e+06 usec per loop
47.46user 0.65system 0:49.05elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (507major+13183minor)pagefaults 0swaps
[alex at lancelot python2.3]$

the issue isn't so much that the elapsed time, per-loop and overall, is
now roughly optimized by a factor of THREE, rather than just TWO as when
we were creating just 10,000 instances -- more significant may be the
fact that VM consumption (as measured by page faults, since Linux doesn't
deign tell us the actual numbers;-) decreases by over 2 times too.  Do
we "*absolutely* have to have* that?  By no means, but since it can be
VERY nice indeed to have, what's the big deal with having it *for those
few classes where __slots__'s restrictions are acceptable, and tens or
hundreds of thousands of instances can well be necessary*?  What is it
that you think makes __slots__ so terrible, compared with many other
possible optimizations?!


So my advice remains: consider __slots__ for classes which may need to
be instantiated a HUGE number of times, if the restrictions that __slots__
imposes can be lived with.  Like for all optimizations, you'll probably
want to measure things, at some level, BEFORE you optimize.  Still,
profiling isn't necessarily all that useful here, since __slots__'s main 
use is in reducing your memory footprint (your working-set) and profilers
aren't very good at showing you where you're spending memory and what the
overall effects of your memory consumption on system performance are (maybe
your program's running fine, but if it hogs more memory than it really needs
it's slowing down _other_ simultaneous activities on the same computer -- 
and whether that matters or not may be dependent on considerations not
directly related to your program, that no profiler, per se, can ever 
capture...;-).  [I really wish I had a good way to find out how much
memory a given container -- together with all of its contents -- is using...
unfortunately, I don't know of any good way to answer that question!!!].


Alex