Memory management

[Marcelo A. Camelo]

WARNING: long post ahead!

I've been researching the feasibility of using Python for the
development of commercial games and I will be giving a presentation
about this subject at the Australian Game Developers Conference by the
end of the month, in Melbourne.

Python memory management happens to be one of the greatest reasons
that game developers resist the ideia of taking Python seriously in
commercial titles, specially for console development. But it seems
that most people complainning used pre 2.0 versions and have never
heard about the Specialized Small Object Allocator. From what I've
learned (mostly from reading PyAlloc source code), python new memory
allocator solves most of the problems faced by game developers.

Bellow is a summary of what I will take to the conference in respect
to Python memory management. Am I on the right track or am I missing
some fundamental truth? I would appreciate any feedback that can
improve the usefulness of the information I will be taking to game
developers.

[summary]
One of the most common complaint I've heard from game developers had
to do with Python's bad memory habbits and the lack of control over
how Python manages memory at the low level. The problem is that, in
Python, every object is allocated dynamically from the heap and
allocations occur very frequently. Even simple ints are objects
allocated from the heap. Relying on standard C malloc and with the
large number of allocations and deallocations taking place even on
simple algorithms, severe memory fragmentation will inevitably occour,
along with the associated performance overhead of fitting memory
requests.

Worse yet is that, differently from CPU time which you can inpect with
a profiler, memory usage is impossible to assess without writting your
own memory manager.

Since Python 2.3, most of these problems have been solved. Python now
comes equipped with the Specialized Small Object Allocator, known as
PyAlloc. PyAlloc has been part of python since version 2.1, but until
version 2.3, it was an optional feature that had to be explicitly
enabled when building python from the sources. Since its first
appearance in 2001, PyAlloc has been tested and improved. In 2003 it
was considered mature enough to become Python's default memory
manager.

PyAlloc implements a modified segmented allocator based on free lists.
This allocation is optimized for frequent allocation and deallocation
of small memory blocks, the most common case in Python programs, being
able to serve memory requests with negligible overhead. Requests for
big blocks are forwarded to the underlying system memory allocator.

By segmenting memory request on the basis of their sizes, PyAlloc
solves the problem of external fragmentation, with the cost of some
negligible internal fragmentation. Also, when built with the
appropriated compilation options, PyAlloc is able to provide
comprehensive memory usage statistics.
[/summary]

Any comments?

Best regards,

Marcelo A. Camelo