[Python-Dev] ast status, memory leaks, etc

[Neal Norwitz]

I lied a bit in my previous status.  I said that the refs used at the
end of a regression test run from a clean state (*) were down to 380k.
 Well if I had remembered to remove all the .pyc's this would have
been true.  Here's the numbers now:

Before AST: r39757 [362766 refs]
Before AST: svn up [356255 refs] 266 OK 31 skipped
clean:             [342367 refs] 267 OK 31 skipped

(*) Before each run I did:  find . -name '*.pyc' | xargs rm

Unless I screwed up again, the first line is from clean at revision
39757 which was just before the AST merge.  The second line was a
selective update of other files that didn't have any relationship to
AST (primarily compile.c and symtable.c).  The last run is after my
recent checkin.

So even with an additional test, we are finishing a regrtest.py run
with less references. I don't know of any constructs which leak
references.

A patch was posted for the free memory read I reported earlier (not
related to AST branch).  It's on SF, I don't know the #.

There are many potential memory leaks in the AST code in error
conditions (hopefully these are only possible when running out of
memory).  It really needs the arena implementation to fix them and get
it right.  There are also still a few printfs in the AST code which
should be changed to SystemErrors.

There are still 2 memory leaks while running the regression tests. 
They show up when running test_fork1 and test_pty.  There may be more,
valgrind crashed on me the last run which was also before I fixed some
of the reference leaks.  It would be great if people could localize
the leaks.

512 bytes in 1 blocks are definitely lost in loss record 319 of 548
   at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
   by 0x433CC4: new_arena (obmalloc.c:500)
   by 0x433EA8: PyObject_Malloc (obmalloc.c:706)
   by 0x43734B: PyString_FromStringAndSize (stringobject.c:74)
   by 0x4655B5: optimize_code (compile.c:957)
   by 0x467B86: makecode (compile.c:4092)
   by 0x467F00: assemble (compile.c:4166)
   by 0x46AA94: compiler_mod (compile.c:1755)
   by 0x46AC8B: PyAST_Compile (compile.c:285)
   by 0x47A870: run_mod (pythonrun.c:1195)
   by 0x47B0E8: PyRun_StringFlags (pythonrun.c:1159)
   by 0x45767A: builtin_eval (bltinmodule.c:589)
   by 0x41684F: PyObject_Call (abstract.c:1777)
   by 0x45EB4B: PyEval_CallObjectWithKeywords (ceval.c:3432)
   by 0x457E4E: builtin_map (bltinmodule.c:938)

1280 bytes in 2 blocks are definitely lost in loss record 383 of 548
   at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
   by 0x433CC4: new_arena (obmalloc.c:500)
   by 0x433EA8: PyObject_Malloc (obmalloc.c:706)
   by 0x4953F3: PyNode_AddChild (node.c:95)
   by 0x495611: shift (parser.c:112)
   by 0x4958F0: PyParser_AddToken (parser.c:244)
   by 0x411704: parsetok (parsetok.c:166)
   by 0x47AE4F: PyParser_ASTFromFile (pythonrun.c:1292)
   by 0x472338: parse_source_module (import.c:777)
   by 0x47262B: load_source_module (import.c:905)
   by 0x4735B3: load_module (import.c:1665)
   by 0x473C4B: import_submodule (import.c:2259)
   by 0x473DE0: load_next (import.c:2079)
   by 0x4741D5: import_module_ex (import.c:1914)
   by 0x474389: PyImport_ImportModuleEx (import.c:1955)