From neal@metaslash.com Tue Apr 1 12:32:39 2003
From: neal@metaslash.com (Neal Norwitz)
Date: Tue, 01 Apr 2003 07:32:39 -0500
Subject: [Compiler-sig] ast branch
Message-ID: <20030401123239.GH16288@epoch.metaslash.com>
The AST branch seems to be making pretty good progress. I just wanted
to collect the known issues so we don't overlap. Brett Cannon had
mentioned that he was interested in the new compiler, so perhaps this
will provide some insight into the status.
Known problems:
* class is not handled
* need more error checking on memory alloc failures, etc.
* use PyOS_snprintf instead of sprintf
* the following code at the module scope core dumps:
if 1:
try:
from posix import _exit
except ImportError:
pass
there is an extra POP_TOP at offset 10 after SETUP_EXCEPT
the initial JUMP_IF_FALSE jumps into the try, not after
The new files are in Python: asdl.c, ast.c newcompile.c.
Python/symtable.c has been modified to use the new AST too.
I'm sure there's a ton of things I forgot or don't know.
Neal
From nas@python.ca Tue Apr 1 14:55:00 2003
From: nas@python.ca (Neil Schemenauer)
Date: Tue, 1 Apr 2003 06:55:00 -0800
Subject: [Compiler-sig] ast branch
In-Reply-To: <20030401123239.GH16288@epoch.metaslash.com>
References: <20030401123239.GH16288@epoch.metaslash.com>
Message-ID: <20030401145500.GA4790@glacier.arctrix.com>
Neal Norwitz wrote:
> * class is not handled
I implemented 'class' at pycon. It might even work. Method calls are
broken. The following code does not work:
class A:
def f(self):
print 'hello'
A().f()
I haven't dug into what's wrong.
Neil
From jeremy@alum.mit.edu Tue Apr 1 15:44:41 2003
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: 01 Apr 2003 10:44:41 -0500
Subject: [Compiler-sig] ast branch
In-Reply-To: <20030401123239.GH16288@epoch.metaslash.com>
References: <20030401123239.GH16288@epoch.metaslash.com>
Message-ID: <1049211881.14152.9.camel@slothrop.zope.com>
On Tue, 2003-04-01 at 07:32, Neal Norwitz wrote:
> The AST branch seems to be making pretty good progress. I just wanted
> to collect the known issues so we don't overlap. Brett Cannon had
> mentioned that he was interested in the new compiler, so perhaps this
> will provide some insight into the status.
I fixed a wild right in assemble_jump_offsets() yesterday. That
eliminate the code dumps I had been seeing during compilation of site.py
and the modules it depends on. I still get a core dump somewhere during
the startup, but the failure is the result of generating bad code. Much
more progress on that front.
We've still got lots of niggling problems. I also fixed a problem with
expressions containing more than one binary operator. 1 + 2 + 3 was
being turned into BinOp(1, +, 2) by the ast converter, instead of
BinOp(BinOp(1, +, 2), + 3).
It's still helpful to try trivial one-line expressions at the
interpreter prompt and see what happens.
Jeremy
From jeremy@alum.mit.edu Tue Apr 1 18:48:34 2003
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: 01 Apr 2003 13:48:34 -0500
Subject: [Compiler-sig] ast branch
In-Reply-To: <20030401123239.GH16288@epoch.metaslash.com>
References: <20030401123239.GH16288@epoch.metaslash.com>
Message-ID: <1049222914.14149.36.camel@slothrop.zope.com>
On Tue, 2003-04-01 at 07:32, Neal Norwitz wrote:
> The AST branch seems to be making pretty good progress. I just wanted
> to collect the known issues so we don't overlap. Brett Cannon had
> mentioned that he was interested in the new compiler, so perhaps this
> will provide some insight into the status.
Here are two important tasks:
- Compute the stack depth from the basic blocks.
- Create a valid lnotab.
I think the former problem is what's prevent us from importing os.py.
Jeremy
From neal@metaslash.com Tue Apr 1 22:25:46 2003
From: neal@metaslash.com (Neal Norwitz)
Date: Tue, 01 Apr 2003 17:25:46 -0500
Subject: [Compiler-sig] ast branch
In-Reply-To: <1049222914.14149.36.camel@slothrop.zope.com>
References: <20030401123239.GH16288@epoch.metaslash.com>
<1049222914.14149.36.camel@slothrop.zope.com>
Message-ID: <20030401222546.GK16288@epoch.metaslash.com>
On Tue, Apr 01, 2003 at 01:48:34PM -0500, Jeremy Hylton wrote:
> On Tue, 2003-04-01 at 07:32, Neal Norwitz wrote:
> > The AST branch seems to be making pretty good progress. I just wanted
> > to collect the known issues so we don't overlap. Brett Cannon had
> > mentioned that he was interested in the new compiler, so perhaps this
> > will provide some insight into the status.
>
> Here are two important tasks:
> - Compute the stack depth from the basic blocks.
> - Create a valid lnotab.
Not sure about the first, but the second would be very helpful when
the interpreter crashes.
I ran with valgrind, there were a few checkins as a result. There
are still some memory leaks and a problem with raise statements IIRC.
==3872== Invalid read of size 2
==3872== at 0x8101E3F: ast_for_expr (Python/ast.c:700)
==3872== by 0x8102D2A: ast_for_flow_stmt (Python/ast.c:1009)
==3872== by 0x810425E: ast_for_stmt (Python/ast.c:1432)
==3872== by 0x81036BF: ast_for_suite (Python/ast.c:1195)
==3872== by 0x810135B: ast_for_funcdef (Python/ast.c:426)
==3872== by 0x810437A: ast_for_stmt (Python/ast.c:1460)
==3872== by 0x8100877: PyAST_FromNode (Python/ast.c:113)
==3872== by 0x80C2483: PyParser_ASTFromFile (Python/pythonrun.c:1102)
==3872== by 0x80B6E63: parse_source_module (Python/import.c:660)
==3872== by 0x80B7115: load_source_module (Python/import.c:782)
==3872== 12 bytes in 1 blocks are definitely lost in loss record 4 of 241
==3872== at 0x40168A70: malloc (vg_clientfuncs.c:100)
==3872== by 0x806C3A7: PyObject_Malloc (Objects/obmalloc.c:854)
==3872== by 0x81002C5: asdl_seq_new (Python/asdl.c:7)
==3872== by 0x8101777: ast_for_listcomp (Python/ast.c:527)
==3872== by 0x81019C6: ast_for_atom (Python/ast.c:579)
==3872== by 0x8102261: ast_for_expr (Python/ast.c:779)
==3872== by 0x81026B8: ast_for_testlist (Python/ast.c:872)
==3872== by 0x8102C8B: ast_for_flow_stmt (Python/ast.c:1000)
==3872== by 0x810425E: ast_for_stmt (Python/ast.c:1432)
==3872== by 0x81036BF: ast_for_suite (Python/ast.c:1195)
==3872== by 0x8103D76: ast_for_except_clause (Python/ast.c:1337)
==3872== by 0x8103F65: ast_for_try_stmt (Python/ast.c:1369)
==3872== by 0x8104366: ast_for_stmt (Python/ast.c:1458)
==3872== by 0x81036BF: ast_for_suite (Python/ast.c:1195)
==3872== by 0x810135B: ast_for_funcdef (Python/ast.c:426)
==3872== 12 bytes in 1 blocks are definitely lost in loss record 6 of 241
==3872== at 0x40168A70: malloc (vg_clientfuncs.c:100)
==3872== by 0x806C3A7: PyObject_Malloc (Objects/obmalloc.c:854)
==3872== by 0x81002C5: asdl_seq_new (Python/asdl.c:7)
==3872== by 0x8102A84: ast_for_exprlist (Python/ast.c:953)
==3872== by 0x81016B7: ast_for_listcomp (Python/ast.c:515)
==3872== by 0x81019C6: ast_for_atom (Python/ast.c:579)
==3872== by 0x8102261: ast_for_expr (Python/ast.c:779)
==3872== by 0x81026B8: ast_for_testlist (Python/ast.c:872)
==3872== by 0x8102C8B: ast_for_flow_stmt (Python/ast.c:1000)
==3872== by 0x810425E: ast_for_stmt (Python/ast.c:1432)
==3872== by 0x81036BF: ast_for_suite (Python/ast.c:1195)
==3872== by 0x8103D76: ast_for_except_clause (Python/ast.c:1337)
==3872== by 0x8103F65: ast_for_try_stmt (Python/ast.c:1369)
==3872== by 0x8104366: ast_for_stmt (Python/ast.c:1458)
==3872== 100 bytes in 5 blocks are definitely lost in loss record 106 of 241
==3872== at 0x40168A70: malloc (vg_clientfuncs.c:100)
==3872== by 0x81059DF: PyNode_New (Parser/node.c:10)
==3872== by 0x80DBEBA: PyParser_New (Parser/parser.c:85)
==3872== by 0x8054F77: parsetok (Parser/parsetok.c:100)
==3872== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81)
==3872== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099)
==3872== by 0x80B6E63: parse_source_module (Python/import.c:660)
==3872== by 0x80B7115: load_source_module (Python/import.c:782)
==3872== by 0x80B7EDC: load_module (Python/import.c:1455)
==3872== by 0x80B95A5: import_submodule (Python/import.c:1994)
From jeremy@alum.mit.edu Tue Apr 1 23:46:35 2003
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: 01 Apr 2003 18:46:35 -0500
Subject: [Compiler-sig] ast branch
In-Reply-To: <20030401222546.GK16288@epoch.metaslash.com>
References: <20030401123239.GH16288@epoch.metaslash.com>
<1049222914.14149.36.camel@slothrop.zope.com>
<20030401222546.GK16288@epoch.metaslash.com>
Message-ID: <1049240794.14146.57.camel@slothrop.zope.com>
I guess I better figure out how to use Valgrind, because it diagnoses
problems faster than I do :-).
The PyObject_Free(mod) fails in a debug build.
file_input containing 38 statements
Debug memory block at address p=0x8144f28:
3688618971 bytes originally requested
The 4 pad bytes at p-4 are not all FORBIDDENBYTE (0xfb):
at p-4: 0x11 *** OUCH
at p-3: 0x00 *** OUCH
at p-2: 0x00 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes
requested
may be bogus, and checking the trailing pad bytes may segfault.
The 4 pad bytes at tail=0xe3f02b03 are Segmentation fault (core
dumped)
Also, the compiler_use_next_block() after a jump is not the right answer
in the long run. The next part means that there is a control flow from
the previous block to the next block. If the previous block ends with a
jump, there's clearly no implicit control flow to the next block. There
must be some other way to fix the problem, but I'm not sure what it is.
Jeremy
From neal@metaslash.com Wed Apr 2 00:02:57 2003
From: neal@metaslash.com (Neal Norwitz)
Date: Tue, 01 Apr 2003 19:02:57 -0500
Subject: [Compiler-sig] ast branch
In-Reply-To: <1049240794.14146.57.camel@slothrop.zope.com>
References: <20030401123239.GH16288@epoch.metaslash.com>
<1049222914.14149.36.camel@slothrop.zope.com>
<20030401222546.GK16288@epoch.metaslash.com>
<1049240794.14146.57.camel@slothrop.zope.com>
Message-ID: <20030402000257.GN16288@epoch.metaslash.com>
On Tue, Apr 01, 2003 at 06:46:35PM -0500, Jeremy Hylton wrote:
> I guess I better figure out how to use Valgrind, because it diagnoses
> problems faster than I do :-).
:-) It's very easy. Build, install, run:
valgrind --show-reachable=yes --num-callers=15 --leak-check=yes ./python
I run under script so I can catch the output, redirecting doesn't work if
there's a crash because valgrind does some funny stuff.
> The PyObject_Free(mod) fails in a debug build.
> file_input containing 38 statements
> Debug memory block at address p=0x8144f28:
> 3688618971 bytes originally requested
> The 4 pad bytes at p-4 are not all FORBIDDENBYTE (0xfb):
> at p-4: 0x11 *** OUCH
> at p-3: 0x00 *** OUCH
> at p-2: 0x00 *** OUCH
> at p-1: 0x00 *** OUCH
> Because memory is corrupted at the start, the count of bytes
> requested
> may be bogus, and checking the trailing pad bytes may segfault.
> The 4 pad bytes at tail=0xe3f02b03 are Segmentation fault (core
> dumped)
Hmmm, I'm also using debug, but don't get the problem when running
under valgrind or not. I have no outstanding changes. But it could
be the read from uninitialized memory is behaving differently on
our boxes too.
> Also, the compiler_use_next_block() after a jump is not the right answer
> in the long run. The next part means that there is a control flow from
> the previous block to the next block. If the previous block ends with a
> jump, there's clearly no implicit control flow to the next block. There
> must be some other way to fix the problem, but I'm not sure what it is.
Ok. I just made it consistent. It was 2 and 2 I think. I think there
is also a lot more memory that stays in use after it is needed.
==7289== 37348 bytes in 7274 blocks are still reachable in loss record 237 of 241
==7289== at 0x40168A70: malloc (vg_clientfuncs.c:100)
==7289== by 0x8055036: parsetok (Parser/parsetok.c:128)
==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81)
==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099)
==7289== by 0x80B6E63: parse_source_module (Python/import.c:660)
==7289== by 0x80B7115: load_source_module (Python/import.c:782)
==7289== by 0x80B7EDC: load_module (Python/import.c:1455)
==7289== by 0x80B95A5: import_submodule (Python/import.c:1994)
==7289== by 0x80B8D47: load_next (Python/import.c:1850)
==7289== by 0x80B869A: import_module_ex (Python/import.c:1701)
==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742)
==7289== by 0x809355C: builtin___import__ (Python/bltinmodule.c:40)
==7289== by 0x80F9A6D: PyCFunction_Call (Objects/methodobject.c:80)
==7289== by 0x80E07EE: PyObject_Call (Objects/abstract.c:1688)
==7289== by 0x80A6E24: PyEval_CallObjectWithKeywords (Python/ceval.c:3026)
==7289==
==7289== 41560 bytes in 2078 blocks are still reachable in loss record 238 of 241
==7289== at 0x40168A70: malloc (vg_clientfuncs.c:100)
==7289== by 0x40168FF0: realloc (vg_clientfuncs.c:265)
==7289== by 0x8105B20: PyNode_AddChild (Parser/node.c:35)
==7289== by 0x80DBF99: shift (Parser/parser.c:112)
==7289== by 0x80DC356: PyParser_AddToken (Parser/parser.c:244)
==7289== by 0x80550AB: parsetok (Parser/parsetok.c:148)
==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81)
==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099)
==7289== by 0x80B6E63: parse_source_module (Python/import.c:660)
==7289== by 0x80B7115: load_source_module (Python/import.c:782)
==7289== by 0x80B7EDC: load_module (Python/import.c:1455)
==7289== by 0x80B95A5: import_submodule (Python/import.c:1994)
==7289== by 0x80B8D47: load_next (Python/import.c:1850)
==7289== by 0x80B869A: import_module_ex (Python/import.c:1701)
==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742)
==7289==
==7289== 124740 bytes in 1576 blocks are still reachable in loss record 239 of 241
==7289== at 0x40169079: realloc (vg_clientfuncs.c:273)
==7289== by 0x8105B20: PyNode_AddChild (Parser/node.c:35)
==7289== by 0x80DC00B: push (Parser/parser.c:126)
==7289== by 0x80DC2E0: PyParser_AddToken (Parser/parser.c:234)
==7289== by 0x80550AB: parsetok (Parser/parsetok.c:148)
==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81)
==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099)
==7289== by 0x80B6E63: parse_source_module (Python/import.c:660)
==7289== by 0x80B7115: load_source_module (Python/import.c:782)
==7289== by 0x80B7EDC: load_module (Python/import.c:1455)
==7289== by 0x80B95A5: import_submodule (Python/import.c:1994)
==7289== by 0x80B8D47: load_next (Python/import.c:1850)
==7289== by 0x80B869A: import_module_ex (Python/import.c:1701)
==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742)
==7289== by 0x809355C: builtin___import__ (Python/bltinmodule.c:40)
==7289==
==7289== 141480 bytes in 1859 blocks are still reachable in loss record 240 of 241
==7289== at 0x40169079: realloc (vg_clientfuncs.c:273)
==7289== by 0x8105B20: PyNode_AddChild (Parser/node.c:35)
==7289== by 0x80DBF99: shift (Parser/parser.c:112)
==7289== by 0x80DC356: PyParser_AddToken (Parser/parser.c:244)
==7289== by 0x80550AB: parsetok (Parser/parsetok.c:148)
==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81)
==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099)
==7289== by 0x80B6E63: parse_source_module (Python/import.c:660)
==7289== by 0x80B7115: load_source_module (Python/import.c:782)
==7289== by 0x80B7EDC: load_module (Python/import.c:1455)
==7289== by 0x80B95A5: import_submodule (Python/import.c:1994)
==7289== by 0x80B8D47: load_next (Python/import.c:1850)
==7289== by 0x80B869A: import_module_ex (Python/import.c:1701)
==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742)
==7289== by 0x809355C: builtin___import__ (Python/bltinmodule.c:40)
==7289==
==7289== 466920 bytes in 23346 blocks are still reachable in loss record 241 of 241
==7289== at 0x40168A70: malloc (vg_clientfuncs.c:100)
==7289== by 0x40168FF0: realloc (vg_clientfuncs.c:265)
==7289== by 0x8105B20: PyNode_AddChild (Parser/node.c:35)
==7289== by 0x80DC00B: push (Parser/parser.c:126)
==7289== by 0x80DC2E0: PyParser_AddToken (Parser/parser.c:234)
==7289== by 0x80550AB: parsetok (Parser/parsetok.c:148)
==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81)
==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099)
==7289== by 0x80B6E63: parse_source_module (Python/import.c:660)
==7289== by 0x80B7115: load_source_module (Python/import.c:782)
==7289== by 0x80B7EDC: load_module (Python/import.c:1455)
==7289== by 0x80B95A5: import_submodule (Python/import.c:1994)
==7289== by 0x80B8D47: load_next (Python/import.c:1850)
==7289== by 0x80B869A: import_module_ex (Python/import.c:1701)
==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742)
From jeremy@alum.mit.edu (Jeremy Hylton) Wed Apr 2 04:30:32 2003
From: jeremy@alum.mit.edu (Jeremy Hylton) (Jeremy Hylton)
Date: Tue, 1 Apr 2003 23:30:32 -0500
Subject: [Compiler-sig] ast branch
In-Reply-To: <20030401222546.GK16288@epoch.metaslash.com>
References: <20030401123239.GH16288@epoch.metaslash.com>
<1049222914.14149.36.camel@slothrop.zope.com>
<20030401222546.GK16288@epoch.metaslash.com>
Message-ID: <16010.26472.219301.756746@slothrop.zope.com>
Thanks for the clue about the Raise statement in ast.c. It was
completely broken, and fixing it provided a lot of progress. Running
"python -v" got to an interpreter prompt:
'import site' failed; traceback:
Traceback (most recent call last):
File "/home/jeremy/src/python-ast/build-pydebug/../Lib/site.py",
line 0, in <module>
File "/home/jeremy/src/python-ast/build-pydebug/../Lib/site.py",
line 0, in <module>
File "/home/jeremy/src/python-ast/build-pydebug/../Lib/os.py", line
0, in <module>
File "/home/jeremy/src/python-ast/build-pydebug/../Lib/os.py",
line 0, in <module>
NameError: name '_Environ' is not defined
Python 2.3a0 (#71, Apr 1 2003, 23:20:02)
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
filename=<stdin>
>>>
It's not perfect, but it's nice to actually survive startup. As we've
noted, it would sure be handy to have some line numbers.
Jeremy
From neal@metaslash.com Thu Apr 3 00:47:47 2003
From: neal@metaslash.com (Neal Norwitz)
Date: Wed, 02 Apr 2003 19:47:47 -0500
Subject: [Compiler-sig] ast branch
In-Reply-To: <16010.26472.219301.756746@slothrop.zope.com>
References: <20030401123239.GH16288@epoch.metaslash.com>
<1049222914.14149.36.camel@slothrop.zope.com>
<20030401222546.GK16288@epoch.metaslash.com>
<16010.26472.219301.756746@slothrop.zope.com>
Message-ID: <20030403004746.GB11263@epoch.metaslash.com>
marshal_write_object() is empty in Python/asdl.c. The only
use is in Python/Python-ast.c for writing a Number:
case Num_kind:
marshal_write_int(buf, off, 10);
marshal_write_object(buf, off, o->v.Num.n);
break;
Not sure what needs to be done here. Do we need to handle
arch sized ints? (ie 4 or 8 bytes). If not, couldn't we
just use marshal_write_int?
Neal
From huey_jiang@yahoo.com Wed Apr 9 05:45:55 2003
From: huey_jiang@yahoo.com (Huey Jiang)
Date: Tue, 8 Apr 2003 21:45:55 -0700 (PDT)
Subject: [Compiler-sig] Chinese version
Message-ID: <20030409044555.74727.qmail@web20001.mail.yahoo.com>
Hi There,
My name is Huey, a python lover. Recently I took my
home visiting to China where IT industry is booming.
However, I noticed that nobody even heard of python:
people are now being trapped by Microsoft and Borland.
Yet, while I was trying to show the people python, I
fell into embaressment: the Unicode of python does not
support Chinese language. What a shame!
I wonder: can you guys make python to support Chinese
language? In fact, I myself am interesting at working
on it if somebody would help me. I guess python is now
supporting non-English languages such as Spanish,
Germny. So, I wish somebody worked on those would give
me helps.
Regards,
Huey
__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - File online, calculators, forms, and more
http://tax.yahoo.com
From jeremy@zope.com Mon Apr 28 19:52:36 2003
From: jeremy@zope.com (Jeremy Hylton)
Date: 28 Apr 2003 14:52:36 -0400
Subject: [Compiler-sig] merged trunk to ast-branch -- line numbers working
Message-ID: <1051555956.17289.122.camel@slothrop.zope.com>
I did some work over the last couple of weeks to get line numbers
working on the ast branch. If you get a traceback, it should not have
meaningful line numbers.
You've probably noticed that I updated the branch from the trunk in
order to get there. This wasn't completely necessary, but was the first
way I thought of to get the interpreter to use co_lnotab instead of
SET_LINENO instructions.
There's now even more code that gets run when the interpreter starts
up. warnings.py was one that caused problems. There are some problems
with the code generated. I haven't tracked down exactly what goes
wrong, but I hope the line numbers help.
Jeremy