Python IS slow ! [was] Re: Python too slow for real world

[Vladimir Marangozov]

[about optimizing Python's memory management]

Christian Tismer wrote:
>
> Thanks a lot. Now I have so many ways to go that I cannot
> decide.

If you promise to think twice before jumping into memory
management, I promise to help you and try to answer all your
questions. ;-)

Hint: before choosing a way to go, (1) look at the stats,
      (2) decide where you want to go, (3) tell us why &
      then go for it.

> Vlad uses explicit bins, dedicated storage. While
> this can be very fast, it is a generalization of Guido's
> pre-allocated arrays for some kinds of objects.

Not quite. It does more than it seems to do.

> Nobody can tell me

Ok. I won't tell you then ;-)

> if this is the way to go,

This is one way to go.

> or if it will under circumstances yield much fragmentation.

Under circumstances, all allocators yield much fragmentation.

> Doug Lea's
> code is more general, since his bins are linked lists of
> free space which can coalesce again. Takes more time and
> less fragmentation.

DL's malloc is a general purpose allocator. It performs well
in the average case. The average case is not Python's case
(see the stats), so when you have 95% of small allocs, where
70% do not exceed 32 bytes, DL's allocator wastes a lot of
space for the links it stores in each block. Besides, as you
noticed, it takes more time.

You'd certainly want to do better, don't you?
You want a special purpose allocator for Python, right?
If yes, DL's strategy is not optimal for you "as is".

> And J.C. Wippler gave me a very fast
> allocator for small objects which uses a power of two
> size scheme, with memory blocks arranged in a way that
> their size is encoded in the address (portable!).

Must be a BSD-malloc variant. I think it wastes too much memory
in the Python case. But it's fast, yes (if you don't start
paging in&out).

> 
> I guess finding out "the right way"(TM) is NP-complete.

Good guess.

> 
> If I knew in advance how long certain objects will live,

The problem wouldn't be NP-complete :-)

> then I would be able to put them into the proper place.
> Does it make sense to do statistics on object lifetime?
> But the only known thing is allocation size.
> ...
>
> But I know it is too late :-)

It's never too late. You've discovered so far 1/4 of the problems a
Python malloc should solve in order to be acceptable ;-)

Still want to dig into memory management?

Remember that whatever you do, you deal with virtual memory, so lots
of things aren't under control. These things are different and specific
to each OS.

Still want to dig into memory management? ;-)

-- 
       Vladimir MARANGOZOV          | Vladimir.Marangozov at inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252