On 2008-02-08 12:23, M.-A. Lemburg wrote:
On 2008-02-08 08:21, Martin v. Löwis wrote:
One of the hopes of having a custom allocator for Python was to be able to get rid off all free lists. For some reason that never happened. Not sure why. People were probably too busy with adding new features to the language at the time ;-) Probably not. It's more that the free lists still outperformed pymalloc.
Something you could try to make PyMalloc perform better for the builtin types is to check the actual size of the allocated PyObjects and then make sure that PyMalloc uses arenas large enough to hold a good quantity of them, e.g. it's possible that the float types fall into the same arena as some other type and thus don't have enough "room" to use as free list. I don't think any improvements can be gained here. PyMalloc carves out pools of 4096 bytes from an arena when it runs out of blocks for a certain size class, and then keeps a linked list of pools of the same size class. So when many float objects get allocated, you'll have a lot of pools of the float type's size class. IOW, PyMalloc has always enough room.
Well, yes, it doesn't run out of memory, but if pymalloc needs to allocate lots of objects of the same size, then performance degrades due to the management overhead involved for checking the free pools as well as creating new arenas as needed.
To reduce this overhead, it may be a good idea to preallocate pools for common sizes and make sure they don't drop under a certain threshold.
Here's a list of a few object sizes in bytes for Python 2.5 on an AMD64 machine:
import mx.Tools mx.Tools.sizeof(int(0)) 24 mx.Tools.sizeof(float(0)) 24
8-bit strings are var objects:
mx.Tools.sizeof(str('')) 40 mx.Tools.sizeof(str('a')) 41
Unicode objects use an external buffer:
mx.Tools.sizeof(unicode('')) 48 mx.Tools.sizeof(unicode('a')) 48
Lists do as well:
mx.Tools.sizeof(list()) 40 mx.Tools.sizeof(list([1,2,3])) 40
Tuples are var objects:
mx.Tools.sizeof(tuple()) 24 mx.Tools.sizeof(tuple([1,2,3])) 48
Old style classes:
class C: pass ... mx.Tools.sizeof(C) 64
New style classes are a lot heavier:
class D(object): pass ... mx.Tools.sizeof(D) 848 mx.Tools.sizeof(type(2)) 848
As you can see, Integers and floats fall into the same pymalloc size class. What's strange in Andrew's result is that both integers and floats use the same free list technique and fall into the same pymalloc size class, yet the results are different.
The only difference that's apparent is that small integers are shared, so depending on the data set used for the test, fewer calls to pymalloc or the free list are made.
Here's the same on a 32-bit machine. Notice the differences:
import mx.Tools mx.Tools.sizeof(int(0)) 12 mx.Tools.sizeof(float(0)) 16 mx.Tools.sizeof(str('')) 24 mx.Tools.sizeof(str('a')) 25 mx.Tools.sizeof(unicode('')) 24 mx.Tools.sizeof(unicode('a')) 24 mx.Tools.sizeof(list()) 20 mx.Tools.sizeof(list([1,2,3])) 20 mx.Tools.sizeof(tuple()) 12 mx.Tools.sizeof(tuple([1,2,3])) 24 class C: pass mx.Tools.sizeof(C) 32 class D(object): pass mx.Tools.sizeof(D) 424 mx.Tools.sizeof(type(2)) 424
Floats use 4 bytes more than integers. However, they still fall into the same pymalloc size class (16 bytes this time around). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 08 2008)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611