[Python-Dev] ast status, memory leaks, etc

[Neal Norwitz]

There's still more clean up work to go, but the current AST is
hopefully much closer to the behaviour before it was checked in. 
There are still a few small memory leaks.

After running the test suite, the total references were around 380k
(down from over 1,000k).  I'm not sure exactly what the total refs
were just before AST was checked in, but I believe it was over 340k. 
So there are likely some more ref leaks that should be investigated. 
It would be good to know the exact number before AST was checked in
and now, minus any new tests.

There is one memory reference error in test_coding:

Invalid read of size 1
   at 0x41304E: tok_nextc (tokenizer.c:876)
   by 0x413874: PyTokenizer_Get (tokenizer.c:1099)
   by 0x411962: parsetok (parsetok.c:124)
   by 0x498D1F: PyParser_ASTFromFile (pythonrun.c:1292)
   by 0x48D79A: load_source_module (import.c:777)
   by 0x48E90F: load_module (import.c:1665)
   by 0x48ED61: import_submodule (import.c:2259)
   by 0x48EF60: load_next (import.c:2079)
   by 0x48F44D: import_module_ex (import.c:1921)
   by 0x48F715: PyImport_ImportModuleEx (import.c:1955)
   by 0x46D090: builtin___import__ (bltinmodule.c:44)
 Address 0x1863E8F6 is 2 bytes before a block of size 8192 free'd
   at 0x11B1BA8A: free (vg_replace_malloc.c:235)
   by 0x4127DB: decoding_fgets (tokenizer.c:167)
   by 0x412F1F: tok_nextc (tokenizer.c:823)
   by 0x413874: PyTokenizer_Get (tokenizer.c:1099)
   by 0x411962: parsetok (parsetok.c:124)
   by 0x498D1F: PyParser_ASTFromFile (pythonrun.c:1292)
   by 0x48D79A: load_source_module (import.c:777)
   by 0x48E90F: load_module (import.c:1665)
   by 0x48ED61: import_submodule (import.c:2259)
   by 0x48EF60: load_next (import.c:2079)
   by 0x48F44D: import_module_ex (import.c:1921)
   by 0x48F715: PyImport_ImportModuleEx (import.c:1955)
   by 0x46D090: builtin___import__ (bltinmodule.c:44)

I had a patch for this somewhere, I'll try to find it.  However, I
only fixed this exact error, there was another path that could still
be problematic.

Most of the memory leaks show up when we are forking in:

test_fork1
test_pty
test_subprocess

Here's what I have so far.  There are probably some more.  It would be
great if someone could try to find and fix these leaks.

n
--
 16 bytes in 1 blocks are definitely lost in loss record 25 of 599
    at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
    by 0x4CA102: alias (Python-ast.c:1066)
    by 0x4CD918: alias_for_import_name (ast.c:2199)
    by 0x4D0C4E: ast_for_stmt (ast.c:2244)
    by 0x4D15E3: PyAST_FromNode (ast.c:234)
    by 0x499078: Py_CompileStringFlags (pythonrun.c:1275)
    by 0x46D6DF: builtin_compile (bltinmodule.c:457)

 56 bytes in 1 blocks are definitely lost in loss record 87 of 599
    at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
    by 0x4C9C92: Name (Python-ast.c:860)
    by 0x4CE4BA: ast_for_expr (ast.c:1222)
    by 0x4D1021: ast_for_stmt (ast.c:1900)
    by 0x4D15E3: PyAST_FromNode (ast.c:234)
    by 0x499078: Py_CompileStringFlags (pythonrun.c:1275)
    by 0x46D6DF: builtin_compile (bltinmodule.c:457)

 112 bytes in 2 blocks are definitely lost in loss record 198 of 674
    at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
    by 0x4C9C92: Name (Python-ast.c:860)
    by 0x4CE4BA: ast_for_expr (ast.c:1222)
    by 0x4D1021: ast_for_stmt (ast.c:1900)
    by 0x4D16D5: PyAST_FromNode (ast.c:275)
    by 0x499078: Py_CompileStringFlags (pythonrun.c:1275)
    by 0x46D6DF: builtin_compile (bltinmodule.c:457)

 56 bytes in 1 blocks are definitely lost in loss record 89 of 599
    at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
    by 0x4C9C92: Name (Python-ast.c:860)
    by 0x4CF3AF: ast_for_arguments (ast.c:650)
    by 0x4D1BFF: ast_for_funcdef (ast.c:830)
    by 0x4D15E3: PyAST_FromNode (ast.c:234)
    by 0x499161: PyRun_StringFlags (pythonrun.c:1275)
    by 0x47B1B2: PyEval_EvalFrameEx (ceval.c:4221)
    by 0x47CCCC: PyEval_EvalCodeEx (ceval.c:2739)
    by 0x47ABCC: PyEval_EvalFrameEx (ceval.c:3657)
    by 0x47CCCC: PyEval_EvalCodeEx (ceval.c:2739)
    by 0x4C27F8: function_call (funcobject.c:550)

 112 bytes in 2 blocks are definitely lost in loss record 189 of 651
    at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
    by 0x4C9C92: Name (Python-ast.c:860)
    by 0x4CE4BA: ast_for_expr (ast.c:1222)
    by 0x4D02F7: ast_for_stmt (ast.c:2028)
    by 0x4D16D5: PyAST_FromNode (ast.c:275)
    by 0x499078: Py_CompileStringFlags (pythonrun.c:1275)
    by 0x46D6DF: builtin_compile (bltinmodule.c:457)

 56 bytes in 1 blocks are definitely lost in loss record 118 of 651
    at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
    by 0x4C9A41: Num (Python-ast.c:751)
    by 0x4CE578: ast_for_expr (ast.c:1237)
    by 0x4CF4ED: ast_for_arguments (ast.c:629)
    by 0x4D1BFF: ast_for_funcdef (ast.c:830)
    by 0x4D15E3: PyAST_FromNode (ast.c:234)
    by 0x499161: PyRun_StringFlags (pythonrun.c:1275)
    by 0x47B1B2: PyEval_EvalFrameEx (ceval.c:4221)
    by 0x47CCCC: PyEval_EvalCodeEx (ceval.c:2739)
    by 0x47ABCC: PyEval_EvalFrameEx (ceval.c:3657)
    by 0x47CCCC: PyEval_EvalCodeEx (ceval.c:2739)
    by 0x4C27F8: function_call (funcobject.c:550)

 112 (56 direct, 56 indirect) bytes in 1 blocks are definitely lost in
loss record 185 of 651
    at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
    by 0x4C97CA: GeneratorExp (Python-ast.c:648)
    by 0x4CEE4F: ast_for_expr (ast.c:1251)
    by 0x4D1021: ast_for_stmt (ast.c:1900)
    by 0x4D16D5: PyAST_FromNode (ast.c:275)
    by 0x499078: Py_CompileStringFlags (pythonrun.c:1275)
    by 0x46D6DF: builtin_compile (bltinmodule.c:457)

 1024 bytes in 1 blocks are definitely lost in loss record 441 of 651
    at 0x11B1AF13: malloc (vg_replace_malloc.c:149)
    by 0x43F8C4: PyObject_Malloc (obmalloc.c:500)
    by 0x4B808F: PyNode_AddChild (node.c:95)
    by 0x4B8386: PyParser_AddToken (parser.c:126)
    by 0x411944: parsetok (parsetok.c:165)
    by 0x499062: Py_CompileStringFlags (pythonrun.c:1271)
    by 0x46D6DF: builtin_compile (bltinmodule.c:457)