[Python-Dev] Memory management in the AST parser & compiler
niko at alum.mit.edu
Wed Nov 16 12:34:29 CET 2005
> Boy am I wanting RAII from C++ for automatic freeing when scope is
> left. Maybe we need to come up with a similar thing, like all memory
> that should be freed once a scope is left must use some special struct
> that stores references to all created memory locally and then a free
> call must be made at all exit points in the function using the special
> struct. Otherwise the pointer is stored in the arena and handled
> en-mass later.
That made sense. I think I'd be opposed to what you describe here
just because I think anything which *requires* that cleanup code be
placed on every function is error prone.
Depending on how much you care about peak memory usage, you do not
necessarily need to worry about freeing pointers as you go. If you
can avoid thinking about it, it makes things much simpler.
If you are concerned with peak memory usage, it gets more
complicated, and you will begin to have greater possibility of user
error. The problem is that dynamically allocated memory often
outlives the stack frame in which it was created. There are several
- If you use ref-counted memory, you can add to the ref count of the
memory which outlives the stack frame; the problem is knowing when to
drop it down again. I think the easiest is to have two lists: one
for memory which will go away quickly, and another for more permanent
memory. The more permanent memory list goes away at the end of the
transform and is hopefully rarely used.
- Another idea is to have trees of arenas: the idea is that when an
arena is created, it is assigned a parent. When an arena is freed,
an arenas in its subtree are also freed. This way you can have one
master arena for exception handling, but if there is some sub-region
where allocations can be grouped together, you create a sub-arena and
free it when that region is complete. Note that if you forget to
free a sub-arena, it will eventually be freed.
There is no one-size-fits-all solution. The right one depends on how
memory is used; but I think all of them are much simpler and less
error prone than tracking individual pointers.
I'd actually be happy to hack on the AST code and try to clean up the
memory usage, assuming that the 2.6 release is far enough out that I
will have time to squeeze it in among the other things I am doing.
More information about the Python-Dev