From bickiia@earlham.edu Wed Feb 9 00:20:04 2000 From: bickiia@earlham.edu (Ian Bicking) Date: Tue, 8 Feb 2000 19:20:04 -0500 Subject: [Compiler-sig] Seperating out the parser Message-ID: <20000208192004.B307@localhost> I'm interested in making Python run in MUQ . I was hoping to take the parser from Python and simply retarget the compiler. I know there are people who would like to do the same for Guile. I've been reading through the source in Python-1.5.2/Parser, but I was hoping to get a little help about where the border between the compiler and the parser is. Just a few pointers as to the functions and datastructures that lie around that border, to give me a place to start from. Thanks -- Ian Bicking / 4869 N. Talman Ave. Apt. G, Chicago, IL 60625 bickiia@earlham.edu / http://www.cs.earlham.edu/~bickiia From jeremy@cnri.reston.va.us Wed Feb 9 17:56:02 2000 From: jeremy@cnri.reston.va.us (Jeremy Hylton) Date: Wed, 9 Feb 2000 12:56:02 -0500 (EST) Subject: [Compiler-sig] Seperating out the parser In-Reply-To: <641773080@toto.iv> Message-ID: <14497.43570.339501.739131@goon.cnri.reston.va.us> Congratulations! You've sent the first message on the compiler-sig. I have been meaning to announce the sig on the main list and get things started here, but I've been busy with other things. >>>>> "IB" == Ian Bicking writes: IB> I'm interested in making Python run in MUQ . IB> I was hoping to take the parser from Python and simply retarget IB> the compiler. I know there are people who would like to do the IB> same for Guile. What is MUQ? I assume it is something that you want to generate code for, but I'm not familiar with it. The Web page says something about virtual worlds, so I'm a bit puzzled. There was some discussion at SPAM8 on generating Scheme code from Python. Personally, I think MzScheme is a better platform than Guile. IB> I've been reading through the source in Python-1.5.2/Parser, but IB> I was hoping to get a little help about where the border between IB> the compiler and the parser is. Just a few pointers as to the IB> functions and datastructures that lie around that border, to IB> give me a place to start from. The border between the parser and compiler is one of the key issues we need to sort out. The Python compiler (Python/compile.c) uses the concrete parse tree generated by the parser. The parser is generated from Grammar/Grammer using pgen. The compiler is a bit hard to read because it's referencing the parse tree nodes used integer indexes. I'm working on a Python compiler written in Python; it generates essentially the same bytecode as compile.c, but it uses an abstract syntax tree and it is a bit more readable. The AST I'm using is the one from Python2C (P2C?). You can get it from the p2c CVS repository. (I can't find a URL for it; perhaps Bill or Greg can point us at it.) The AST is produced using the builtin Python parser; the transformer module in p2c gets a parse tree using the parser module and converts into an AST. I think this is a good starting point for developing a compiler for a different target. I expect to propose some changes to the AST when I'm done with the compiler. It looks like nodes like Add, Mul, & Sub should be combined into a BinaryOp node, and UnaryAdd, UnarySub, etc. should become UnaryOp. I'm going to wait until I've got a working version before I make too much noise about these changes though. I also plan to write a new parser for JPython, using ANTLR to directly produce the AST. I imagine that will also improve my perspective on the right AST. I hope to have a snapshot of my code available from the Python CVS server early next week. Jeremy From bickiia@earlham.edu Sat Feb 12 19:43:59 2000 From: bickiia@earlham.edu (Ian Bicking) Date: Sat, 12 Feb 2000 14:43:59 -0500 Subject: [Compiler-sig] Seperating out the parser In-Reply-To: <14497.43570.339501.739131@goon.cnri.reston.va.us>; from jeremy@cnri.reston.va.us on Wed, Feb 09, 2000 at 12:56:02PM -0500 References: <641773080@toto.iv> <14497.43570.339501.739131@goon.cnri.reston.va.us> Message-ID: <20000212144359.A402@localhost> I was disapointed when I didn't get a response, then it turns out I had just lost the email in the shuffle. Well, that's a pleasant surprise. Anyway... On Wed, Feb 09, 2000 at 12:56:02PM -0500, Jeremy Hylton wrote: > Congratulations! You've sent the first message on the compiler-sig. > I have been meaning to announce the sig on the main list and get > things started here, but I've been busy with other things. > > >>>>> "IB" == Ian Bicking writes: > > IB> I'm interested in making Python run in MUQ . > IB> I was hoping to take the parser from Python and simply retarget > IB> the compiler. I know there are people who would like to do the > IB> same for Guile. > > What is MUQ? I assume it is something that you want to generate code > for, but I'm not familiar with it. The Web page says something about > virtual worlds, so I'm a bit puzzled. The intention of MUQ is something for making MUDs, but really it's a persistent programming environment which could be used for MUDs, but is considerably more general than that. It already has a few syntaxes that compile to its bytecodes -- forth and lisp being the most natural translations. The C translation doesn't, well, act much like C at all. Which made me think that Python syntax for it would almost have Python semantics, and provide something more C like than lisp or forth. > There was some discussion at SPAM8 on generating Scheme code from > Python. Personally, I think MzScheme is a better platform than > Guile. That's possible -- I only mentioned Guile because I know there are people involved with Guile who are interested in this. Are there archives for the SPAM8 list that I can read? > IB> I've been reading through the source in Python-1.5.2/Parser, but > IB> I was hoping to get a little help about where the border between > IB> the compiler and the parser is. Just a few pointers as to the > IB> functions and datastructures that lie around that border, to > IB> give me a place to start from. > > The border between the parser and compiler is one of the key issues > we need to sort out. The Python compiler (Python/compile.c) uses the > concrete parse tree generated by the parser. The parser is generated > from Grammar/Grammer using pgen. The compiler is a bit hard to read > because it's referencing the parse tree nodes used integer indexes. I'm afraid my knowledge of parsing/compiling is a little narrow, so I'm not familiar with the distinction between concrete and abstract parse trees. But then, I've always tried to fudge my way through parsing, and this is just another case of that since I'm really just hoping to figure out how to *use* the already exist parser(s) rather than implement one. I guess I should learn a little more about these things. [and, having just looked up the definition, I can see why an abstract syntax would be much more useful] The output from the parser module in Python creates something more of a concrete parse tree, doesn't it? It's called an AST, but it seems to include things like NEWLINE, which seems rather concrete. > I'm working on a Python compiler written in Python; it generates > essentially the same bytecode as compile.c, but it uses an abstract > syntax tree and it is a bit more readable. The AST I'm using is the > one from Python2C (P2C?). You can get it from the p2c CVS repository. > (I can't find a URL for it; perhaps Bill or Greg can point us at it.) > The AST is produced using the builtin Python parser; the transformer > module in p2c gets a parse tree using the parser module and converts > into an AST. I think this is a good starting point for developing a > compiler for a different target. http://lima.mudlib.org/~rassilon/p2c/ I'll definately be looking into this. What I'd idealy like to do is to seperate off the parser and make it possible to compile it into a little C program that can print out the parse tree. That way I'd know I had it figured out enough to use, and I think some other people who'd like to use this (as I mentioned with Guile) would find that an easy place to start from. > I expect to propose some changes to the AST when I'm done with the > compiler. It looks like nodes like Add, Mul, & Sub should be combined > into a BinaryOp node, and UnaryAdd, UnarySub, etc. should become > UnaryOp. I'm going to wait until I've got a working version before I > make too much noise about these changes though. I also plan to write > a new parser for JPython, using ANTLR to directly produce the AST. I > imagine that will also improve my perspective on the right AST. > > I hope to have a snapshot of my code available from the Python CVS > server early next week. -- Ian Bicking / 4869 N. Talman Ave. Apt. G, Chicago, IL 60625 bickiia@earlham.edu / http://www.cs.earlham.edu/~bickiia From Fredrik Lundh" <14497.43570.339501.739131@goon.cnri.reston.va.us> <20000212144359.A402@localhost> Message-ID: <00f101bf760d$3519d100$34aab5d4@hagrid> Ian Bicking wrote: > > There was some discussion at SPAM8 on generating Scheme code from > > Python. Personally, I think MzScheme is a better platform than > > Guile. >=20 > That's possible -- I only mentioned Guile because I know there are > people involved with Guile who are interested in this. Are there=20 > archives for the SPAM8 list that I can read? I think jeremy was referring to the eighth international python conference (ipc8 aka. spam8). so I'm afraid the answer is "no, not really". From jeremy@cnri.reston.va.us Sun Feb 13 17:38:20 2000 From: jeremy@cnri.reston.va.us (Jeremy Hylton) Date: Sun, 13 Feb 2000 12:38:20 -0500 (EST) Subject: [Compiler-sig] Seperating out the parser In-Reply-To: <00f101bf760d$3519d100$34aab5d4@hagrid> References: <641773080@toto.iv> <14497.43570.339501.739131@goon.cnri.reston.va.us> <20000212144359.A402@localhost> <00f101bf760d$3519d100$34aab5d4@hagrid> Message-ID: <14502.60428.128341.622840@bitdiddle.cnri.reston.va.us> >>>>> "FL" == Fredrik Lundh writes: FL> Ian Bicking wrote: >> > There was some discussion at SPAM8 on generating Scheme code >> from > Python. Personally, I think MzScheme is a better platform >> than > Guile. >> >> That's possible -- I only mentioned Guile because I know there >> are people involved with Guile who are interested in this. Are >> there archives for the SPAM8 list that I can read? FL> I think jeremy was referring to the eighth international python FL> conference (ipc8 aka. spam8). so I'm afraid the answer is "no, FL> not really". Some of the discussion may have made into Ping's session notes. Check the compiler-sig page. There wasn't a lot of content, merely the observation that it might be a useful project. Jeremy From jeremy@cnri.reston.va.us Sun Feb 13 17:39:41 2000 From: jeremy@cnri.reston.va.us (Jeremy Hylton) Date: Sun, 13 Feb 2000 12:39:41 -0500 (EST) Subject: [Compiler-sig] Seperating out the parser In-Reply-To: <20000212144359.A402@localhost> References: <641773080@toto.iv> <14497.43570.339501.739131@goon.cnri.reston.va.us> <20000212144359.A402@localhost> Message-ID: <14502.60509.898080.599213@bitdiddle.cnri.reston.va.us> >>>>> "IB" == Ian Bicking writes: IB> I've been reading through the source in Python-1.5.2/Parser, but IB> I was hoping to get a little help about where the border between IB> the compiler and the parser is. Just a few pointers as to the IB> functions and datastructures that lie around that border, to IB> give me a place to start from. >> The border between the parser and compiler is one of the key >> issues we need to sort out. The Python compiler >> (Python/compile.c) uses the concrete parse tree generated by the >> parser. The parser is generated from Grammar/Grammer using pgen. >> The compiler is a bit hard to read because it's referencing the >> parse tree nodes used integer indexes. IB> I'm afraid my knowledge of parsing/compiling is a little narrow, IB> so I'm not familiar with the distinction between concrete and IB> abstract parse trees. But then, I've always tried to fudge my IB> way through parsing, and this is just another case of that since IB> I'm really just hoping to figure out how to *use* the already IB> exist parser(s) rather than implement one. I guess I should IB> learn a little more about these things. [and, having just IB> looked up the definition, I can see why an abstract syntax would IB> be much more useful] IB> The output from the parser module in Python creates something IB> more of a concrete parse tree, doesn't it? It's called an AST, IB> but it seems to include things like NEWLINE, which seems rather IB> concrete. You've got it exactly. The concrete syntax includes things like newlines and parens. The abstract syntax does not. Jeremy From dheise@debitel.net Sun Feb 13 19:26:46 2000 From: dheise@debitel.net (Dirk Heise) Date: Sun, 13 Feb 2000 20:26:46 +0100 Subject: [Compiler-sig] Python in Python - Great Idea! Message-ID: <20000213192758.19565.qmail@psmtp1.dnsg.net> I just discovered this list. Great thing! Writing a Python implementation in Python is the coolest idea i heard of for a long time! One question: Will it offer the possibilities of Stackless Python? That means, "freeze" the interpreter state at any time? Dirk From skip@mojam.com (Skip Montanaro) Tue Feb 15 19:50:29 2000 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 15 Feb 2000 13:50:29 -0600 (CST) Subject: [Compiler-sig] Availability of my peephole optimizer In-Reply-To: <14505.40262.109290.196694@weyr.cnri.reston.va.us> References: <14505.36861.554432.428111@weyr.cnri.reston.va.us> <14505.38665.827583.106215@beluga.mojam.com> <14505.40262.109290.196694@weyr.cnri.reston.va.us> Message-ID: <14505.44549.633064.672915@beluga.mojam.com> [ moved from the python-checkins list to compiler-sig and re-subjected ... ] Fred> You're right, it *should* be there. 5+.5j should be a constant as Fred> well. Funny thing. My peepholer does that too. Jeremy sent me a message earlier asking if my peephole optimizer was available as something other than a patch. I'll try and whip it into a bit better shape and bring it into line with the current CVS code and make it available as a more-or-less package. In places where existing code is modified (e.g. ceval.c, compile.c, dis.py) I think it makes more sense to distribute a patch than a new file, however, since patches stand a better chance of being robust in the face of other changes. Is Greg Ward or other distutils expert seeing this (how about Greg Stein...)? If so, would you care to comment on the ability of distutils to handle patches to the core distribution? Is it worth considering making this stuff available via distutils at this point given that it requires mods to so many files in the core distribution? Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/ "Languages that change by catering to the tastes of non-users tend not to do so well." - Doug Landauer From fdrake@acm.org Tue Feb 15 20:11:09 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 15 Feb 2000 15:11:09 -0500 (EST) Subject: [Compiler-sig] Re: Availability of my peephole optimizer In-Reply-To: <14505.44549.633064.672915@beluga.mojam.com> References: <14505.36861.554432.428111@weyr.cnri.reston.va.us> <14505.38665.827583.106215@beluga.mojam.com> <14505.40262.109290.196694@weyr.cnri.reston.va.us> <14505.44549.633064.672915@beluga.mojam.com> Message-ID: <14505.45789.893775.897920@weyr.cnri.reston.va.us> Skip Montanaro writes: > Is Greg Ward or other distutils expert seeing this (how about Greg > Stein...)? If so, would you care to comment on the ability of distutils to > handle patches to the core distribution? Is it worth considering making > this stuff available via distutils at this point given that it requires mods > to so many files in the core distribution? I'd say that it doesn't make sense. Perhaps there could be two packages: one for modifications to the core and one for everything else. But that only makes sense if "everything else" is usable without the core changes. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From skip@mojam.com (Skip Montanaro) Tue Feb 15 21:16:11 2000 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 15 Feb 2000 15:16:11 -0600 (CST) Subject: [Compiler-sig] Re: Availability of my peephole optimizer In-Reply-To: <14505.45789.893775.897920@weyr.cnri.reston.va.us> References: <14505.36861.554432.428111@weyr.cnri.reston.va.us> <14505.38665.827583.106215@beluga.mojam.com> <14505.40262.109290.196694@weyr.cnri.reston.va.us> <14505.44549.633064.672915@beluga.mojam.com> <14505.45789.893775.897920@weyr.cnri.reston.va.us> Message-ID: <14505.49691.443314.160179@beluga.mojam.com> Fred> I'd say that it doesn't make sense. Perhaps there could be two Fred> packages: one for modifications to the core and one for everything Fred> else. But that only makes sense if "everything else" is usable Fred> without the core changes. Unfortunately, without the hooks in the core, it doesn't make a lot of sense right now. If I assume the "new" module is always going to be around I could whack things to use it instead of requiring hooks in compile.c. I suspect Guido would like this approach better. Skip From fdrake@acm.org Tue Feb 15 21:18:50 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 15 Feb 2000 16:18:50 -0500 (EST) Subject: [Compiler-sig] Re: Availability of my peephole optimizer In-Reply-To: <14505.49691.443314.160179@beluga.mojam.com> References: <14505.36861.554432.428111@weyr.cnri.reston.va.us> <14505.38665.827583.106215@beluga.mojam.com> <14505.40262.109290.196694@weyr.cnri.reston.va.us> <14505.44549.633064.672915@beluga.mojam.com> <14505.45789.893775.897920@weyr.cnri.reston.va.us> <14505.49691.443314.160179@beluga.mojam.com> Message-ID: <14505.49850.81931.224550@weyr.cnri.reston.va.us> Skip Montanaro writes: > Unfortunately, without the hooks in the core, it doesn't make a lot of sense > right now. If I assume the "new" module is always going to be around I > could whack things to use it instead of requiring hooks in compile.c. I > suspect Guido would like this approach better. The new module is now on by default. Since pickle uses it, I don't expect that to change any time soon. ;) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From jeremy@cnri.reston.va.us Tue Feb 15 22:19:57 2000 From: jeremy@cnri.reston.va.us (Jeremy Hylton) Date: Tue, 15 Feb 2000 17:19:57 -0500 (EST) Subject: [Compiler-sig] FYI: python/nondist/src/Compiler Message-ID: <14505.53517.545008.521133@goon.cnri.reston.va.us> I've made my current codebase available as part of the Python CVS tree, under the nondist branch. The current code implements most of a Python source code to PyVM compiler. The code is definitely rough; "refactoring" is the buzzword that needs to be applied. You can browse the CVS tree from http://cvs.python.org Jeremy From gstein@lyra.org Wed Feb 16 00:27:48 2000 From: gstein@lyra.org (Greg Stein) Date: Tue, 15 Feb 2000 16:27:48 -0800 (PST) Subject: [Compiler-sig] Re: [Python-checkins] CVS: /python/nondist/src/Compiler compile.py,1.13,1.14 In-Reply-To: <200002152345.SAA15655@goon.cnri.reston.va.us> Message-ID: Yay! I'm glad that you chose an "emit" model for pumping out the opcodes. I'm using the same model for another VM and it is much superior to the "return the code" model used by P2C's C source generator. Coolness. Bill and I want to change P2C's C gen over to that, but it will be a big change. I'd also hope that we can find some common utilities and stuff between the PVM generator and the C generator. In particular: namespace abstractions/handling would be nice. Cheers, -g On Tue, 15 Feb 2000, Jeremy Hylton wrote: > Update of /projects/cvsroot//python/nondist/src/Compiler > In directory goon.cnri.reston.va.us:/home/jhylton/python/nondist/src/Compiler > > Modified Files: > compile.py > Log Message: > finish first impl of code generator > > add support for nodes TryExcept, TryFinally, Sliceobj > fix visitSubscript to properly handle x[a,b,c] > > > > > > Index: compile.py > =================================================================== > RCS file: /projects/cvsroot//python/nondist/src/Compiler/compile.py,v > retrieving revision 1.13 > retrieving revision 1.14 > diff -C2 -r1.13 -r1.14 > *** compile.py 2000/02/14 23:57:56 1.13 > --- compile.py 2000/02/15 23:45:26 1.14 > *************** > *** 156,167 **** > > class CodeGenerator: > ! """TODO > > - EmptyNode > - Sliceobj > - Tryexcept > - Tryfinally > - """ > - > OPTIMIZED = 1 > > --- 156,161 ---- > > class CodeGenerator: > ! """Generate bytecode for the Python VM""" > > OPTIMIZED = 1 > > *************** > *** 426,429 **** > --- 420,480 ---- > self.emit('JUMP_ABSOLUTE', l.startAnchor) > > + def visitTryExcept(self, node): > + # XXX need to figure out exactly what is on the stack when an > + # exception is raised and the first handler is checked > + handlers = StackRef() > + end = StackRef() > + if node.else_: > + lElse = StackRef() > + else: > + lElse = end > + self.emit('SET_LINENO', node.lineno) > + self.emit('SETUP_EXCEPT', handlers) > + self.visit(node.body) > + self.emit('POP_BLOCK') > + self.emit('JUMP_FORWARD', lElse) > + handlers.bind(self.code.getCurInst()) > + > + last = len(node.handlers) - 1 > + for i in range(len(node.handlers)): > + expr, target, body = node.handlers[i] > + if hasattr(expr, 'lineno'): > + self.emit('SET_LINENO', expr.lineno) > + if expr: > + self.emit('DUP_TOP') > + self.visit(expr) > + self.emit('COMPARE_OP', "exception match") > + next = StackRef() > + self.emit('JUMP_IF_FALSE', next) > + self.emit('POP_TOP') > + self.emit('POP_TOP') > + if target: > + self.visit(target) > + else: > + self.emit('POP_TOP') > + self.emit('POP_TOP') > + self.visit(body) > + self.emit('JUMP_FORWARD', end) > + next.bind(self.code.getCurInst()) > + self.emit('POP_TOP') > + self.emit('END_FINALLY') > + if node.else_: > + lElse.bind(self.code.getCurInst()) > + self.visit(node.else_) > + end.bind(self.code.getCurInst()) > + return 1 > + > + def visitTryFinally(self, node): > + final = StackRef() > + self.emit('SET_LINENO', node.lineno) > + self.emit('SETUP_FINALLY', final) > + self.visit(node.body) > + self.emit('POP_BLOCK') > + self.emit('LOAD_CONST', None) > + final.bind(self.code.getCurInst()) > + self.visit(node.final) > + self.emit('END_FINALLY') > + return 1 > + > def visitCompare(self, node): > """Comment from compile.c follows: > *************** > *** 493,500 **** > def visitSubscript(self, node): > self.visit(node.expr) > ! for sub in node.subs[:-1]: > self.visit(sub) > ! self.emit('BINARY_SUBSCR') > ! self.visit(node.subs[-1]) > if node.flags == 'OP_APPLY': > self.emit('BINARY_SUBSCR') > --- 544,551 ---- > def visitSubscript(self, node): > self.visit(node.expr) > ! for sub in node.subs: > self.visit(sub) > ! if len(node.subs) > 1: > ! self.emit('BUILD_TUPLE', len(node.subs)) > if node.flags == 'OP_APPLY': > self.emit('BINARY_SUBSCR') > *************** > *** 503,507 **** > elif node.flags == 'OP_DELETE': > self.emit('DELETE_SUBSCR') > ! > return 1 > > --- 554,558 ---- > elif node.flags == 'OP_DELETE': > self.emit('DELETE_SUBSCR') > ! print > return 1 > > *************** > *** 522,527 **** > self.emit('DELETE_SLICE+%d' % slice) > else: > ! print node.flags > raise > return 1 > > --- 573,585 ---- > self.emit('DELETE_SLICE+%d' % slice) > else: > ! print "weird slice", node.flags > raise > + return 1 > + > + def visitSliceobj(self, node): > + for child in node.nodes: > + print child > + self.visit(child) > + self.emit('BUILD_SLICE', len(node.nodes)) > return 1 -- Greg Stein, http://www.lyra.org/ From mwh21@cam.ac.uk Wed Feb 16 07:59:08 2000 From: mwh21@cam.ac.uk (Michael Hudson) Date: 16 Feb 2000 07:59:08 +0000 Subject: [Compiler-sig] FYI: python/nondist/src/Compiler In-Reply-To: Jeremy Hylton's message of "Tue, 15 Feb 2000 17:19:57 -0500 (EST)" References: <14505.53517.545008.521133@goon.cnri.reston.va.us> Message-ID: Jeremy Hylton writes: > I've made my current codebase available as part of the Python CVS > tree, under the nondist branch. The current code implements most of a > Python source code to PyVM compiler. The code is definitely rough; > "refactoring" is the buzzword that needs to be applied. > > You can browse the CVS tree from http://cvs.python.org > And one might get this code how? I seem to be having CVS-driving problems. Cheers, M. From gstein@lyra.org Wed Feb 16 08:47:52 2000 From: gstein@lyra.org (Greg Stein) Date: Wed, 16 Feb 2000 00:47:52 -0800 (PST) Subject: [Compiler-sig] FYI: python/nondist/src/Compiler In-Reply-To: Message-ID: On 16 Feb 2000, Michael Hudson wrote: > Jeremy Hylton writes: > > I've made my current codebase available as part of the Python CVS > > tree, under the nondist branch. The current code implements most of a > > Python source code to PyVM compiler. The code is definitely rough; > > "refactoring" is the buzzword that needs to be applied. > > > > You can browse the CVS tree from http://cvs.python.org > > > > And one might get this code how? I seem to be having CVS-driving > problems. Follow the instructions on http://www.pythonorg/download/cvs.html, but on the "cvs ... co python" line, use "cvs ... co python/nondist". Using the directions as-is from that page will get Jeremy's work AND the full Python source distribution. The python/nondist checkout limits things a bit. If you don't have CVS or don't want to use it, then you can grab a tarball of the repository from http://starship.python.net/crew/da/pythondists/. Oh! Hrm. I just checked that... it appears that the tarball doesn't (yet) include the "nondist" branch of the CVS repository. Maybe David Ascher can look into that. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mwh21@cam.ac.uk Wed Feb 16 09:24:14 2000 From: mwh21@cam.ac.uk (Michael Hudson) Date: 16 Feb 2000 09:24:14 +0000 Subject: [Compiler-sig] FYI: python/nondist/src/Compiler In-Reply-To: Greg Stein's message of "Wed, 16 Feb 2000 00:47:52 -0800 (PST)" References: Message-ID: Greg Stein writes: > On 16 Feb 2000, Michael Hudson wrote: > > Jeremy Hylton writes: > > > I've made my current codebase available as part of the Python CVS > > > tree, under the nondist branch. The current code implements most of a > > > Python source code to PyVM compiler. The code is definitely rough; > > > "refactoring" is the buzzword that needs to be applied. > > > > > > You can browse the CVS tree from http://cvs.python.org > > > > > > > And one might get this code how? I seem to be having CVS-driving > > problems. > > Follow the instructions on http://www.pythonorg/download/cvs.html, but on > the "cvs ... co python" line, use "cvs ... co python/nondist". Using the > directions as-is from that page will get Jeremy's work AND the full Python > source distribution. The python/nondist checkout limits things a bit. Ah, cheers. I was trying to be too clever as usual, updating my python/dist tree to include the nondist bit as well; got it now, thanks. Cheers, M. From fdrake@acm.org Wed Feb 16 14:41:02 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 16 Feb 2000 09:41:02 -0500 (EST) Subject: [Compiler-sig] Name for Jeremy's code Message-ID: <14506.46846.128777.715634@weyr.cnri.reston.va.us> Jeremy needs a working name for his compiler. I propose jpyc -- "Jeremy's Python Compiler". I wanted to check it out parallel to my Python checkout (which is just the dist/src/ component of the Python tree), and needed a name for the directory that didn't start with "py". (Wouldn't want to type to many characters before using the shells filename completion, now! Those keyboard habits are hard to change!). -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From gstein@lyra.org Wed Feb 16 14:51:14 2000 From: gstein@lyra.org (Greg Stein) Date: Wed, 16 Feb 2000 06:51:14 -0800 (PST) Subject: [Compiler-sig] Name for Jeremy's code In-Reply-To: <14506.46846.128777.715634@weyr.cnri.reston.va.us> Message-ID: On Wed, 16 Feb 2000, Fred L. Drake, Jr. wrote: > Jeremy needs a working name for his compiler. I propose jpyc -- > "Jeremy's Python Compiler". > I wanted to check it out parallel to my Python checkout (which is > just the dist/src/ component of the Python tree), and needed a name > for the directory that didn't start with "py". (Wouldn't want to type > to many characters before using the shells filename completion, now! > Those keyboard habits are hard to change!). Same here... but I just called it "compiler" :-) The leading j is confusing with JPython. I say follow the P2C pattern and call it P2P :-) -g -- Greg Stein, http://www.lyra.org/ From jeremy@cnri.reston.va.us Wed Feb 16 15:53:38 2000 From: jeremy@cnri.reston.va.us (Jeremy Hylton) Date: Wed, 16 Feb 2000 10:53:38 -0500 (EST) Subject: [Compiler-sig] FYI: python/nondist/src/Compiler In-Reply-To: References: <14505.53517.545008.521133@goon.cnri.reston.va.us> Message-ID: <14506.51202.292148.160731@bitdiddle.cnri.reston.va.us> There are instructions at http://www.python.org/download/cvs.html. I hope they're sufficient, because I've never used the remote CVS server. Jeremy From bickiia@earlham.edu Wed Feb 16 17:10:35 2000 From: bickiia@earlham.edu (Ian Bicking) Date: Wed, 16 Feb 2000 12:10:35 -0500 Subject: [Compiler-sig] Name for Jeremy's code In-Reply-To: ; from gstein@lyra.org on Wed, Feb 16, 2000 at 06:51:14AM -0800 References: <14506.46846.128777.715634@weyr.cnri.reston.va.us> Message-ID: <20000216121035.A361@localhost> On Wed, Feb 16, 2000 at 06:51:14AM -0800, Greg Stein wrote: > On Wed, 16 Feb 2000, Fred L. Drake, Jr. wrote: > > Jeremy needs a working name for his compiler. I propose jpyc -- > > "Jeremy's Python Compiler". > > I wanted to check it out parallel to my Python checkout (which is > > just the dist/src/ component of the Python tree), and needed a name > > for the directory that didn't start with "py". (Wouldn't want to type > > to many characters before using the shells filename completion, now! > > Those keyboard habits are hard to change!). > > Same here... but I just called it "compiler" :-) > > The leading j is confusing with JPython. I say follow the P2C pattern and > call it P2P :-) How about Metapyc? I don't think it's really a valid use of "meta" but it comes out as a real word, which is nice. -- Ian Bicking / 4869 N. Talman Ave. Apt. G, Chicago, IL 60625 bickiia@earlham.edu / http://www.cs.earlham.edu/~bickiia From jeremy@cnri.reston.va.us Mon Feb 21 16:21:24 2000 From: jeremy@cnri.reston.va.us (Jeremy Hylton) Date: Mon, 21 Feb 2000 11:21:24 -0500 (EST) Subject: [Compiler-sig] FYI: python/nondist/src/Compiler In-Reply-To: References: <14505.53517.545008.521133@goon.cnri.reston.va.us> Message-ID: <14513.26116.110383.957967@goon.cnri.reston.va.us> >>>>> "MH" == Mark Hammond writes: MH> Ideally p2c and this compiler could share some common framework. MH> Greg also mentioned this. Ive got another project that was MH> based on P2C that Im working fulltime on (and Greg is helping) - MH> so that makes it 3 "compilers" we have going. I bet that Do tell! What is this other project? At devday, Paul Dubois suggested generating C++ code using CXX instead of straight C code; that struck me as an excellent evolutionary path for p2c. MH> someone from the Types-SIG will jump on this for their MH> next-greatest type-inferencer :-) That's my plan :-). MH> So, I had a think about reusing some of this framework for other MH> compilers, and come to the conclusion that not as much as I had MH> hoped can be shared ( But still enough that it makes it MH> worthwhile. I can see the P2C directory stuff being shared, MH> along with the basic tree stuff you have. It depends what the next steps for any of these compilers are. The code I have checked in is pretty primitive; there isn't much to share, just because there isn't much there at all. I hope to refactor the pyassem module and revise the code generator a bit. Instead of emitting straight-line code, with refs -- misnamed StackRefs -- to other locations in the instruction stream, I would like to generate code blocks that contain references to other blocks. This intermediate representation would be more amenable to analysis and still easy to generate bytecode from. [Mark's suggested re-packaging:] compiler\__init__.py # Contains only: from compiler import compile compiler\compiler.py # Existing compile.py, less the first 1/4. compiler\tools\__init__.py # empty file compiler\tools\transformer.py, ast.py, etc compiler\tools\visitor.py # first 1/4 of existing compile.py MH> If you just change the COPYRIGHT file to reflect that it was MH> taken from the P2C project and renamed to "compiler.tools" MH> everyone should be happy I would think... Sounds good to me; makes sense to move the visitor code to compiler.tools. This change okay with you, Greg & Bill? MH> Greg suggested we can reuse some of the name-lookup code, but Im MH> not too sure we can. What Jeremy has is a faithful MH> implementation of the Python name bindings scheme, which is MH> perfect. But any optimizing compiler (which includes P2C) and MH> any type inference systems will need to go a little bit deeper MH> than that - eg, P2C can optimize away certain builtin functions, MH> but to do that it needs to make the distinction between "global MH> cos Ive never seen it before" vs "global and Ive seen an MH> assignment". I had hoped to do some dataflow analysis, but won't have time to work on it in the near future. It's also possible that Jon Riehl will release his dataflow code soon, which might make sense to incorporate as part of the standard tool suite. Jeremy From mhammond@skippinet.com.au Mon Feb 21 05:31:53 2000 From: mhammond@skippinet.com.au (Mark Hammond) Date: Mon, 21 Feb 2000 16:31:53 +1100 Subject: [Compiler-sig] FYI: python/nondist/src/Compiler In-Reply-To: <14505.53517.545008.521133@goon.cnri.reston.va.us> Message-ID: > I've made my current codebase available as part of the Python CVS > tree, under the nondist branch. The current code implements most of a > Python source code to PyVM compiler. The code is definitely rough; > "refactoring" is the buzzword that needs to be applied. This is definately cool. Good job. Ideally p2c and this compiler could share some common framework. Greg also mentioned this. Ive got another project that was based on P2C that Im working fulltime on (and Greg is helping) - so that makes it 3 "compilers" we have going. I bet that someone from the Types-SIG will jump on this for their next-greatest type-inferencer :-) So, I had a think about reusing some of this framework for other compilers, and come to the conclusion that not as much as I had hoped can be shared ( But still enough that it makes it worthwhile. I can see the P2C directory stuff being shared, along with the basic tree stuff you have. So Im proposing that we have a bit of a fiddle with the CVS tree. Rename the package from "p2c" to "tools" - and the compiler itself can also be a package. eg: compiler\__init__.py # Contains only: from compiler import compile compiler\compiler.py # Existing compile.py, less the first 1/4. compiler\tools\__init__.py # empty file compiler\tools\transformer.py, ast.py, etc compiler\tools\visitor.py # first 1/4 of existing compile.py Ultimately, the idea is that: >>> import compiler >>> compiler.compile(code) can be used in place of the built-in compiler. Tools like "p2c" etc can simply use "compiler.tools", and gain access to the core AST and tree-walking code. If you just change the COPYRIGHT file to reflect that it was taken from the P2C project and renamed to "compiler.tools" everyone should be happy I would think... Greg suggested we can reuse some of the name-lookup code, but Im not too sure we can. What Jeremy has is a faithful implementation of the Python name bindings scheme, which is perfect. But any optimizing compiler (which includes P2C) and any type inference systems will need to go a little bit deeper than that - eg, P2C can optimize away certain builtin functions, but to do that it needs to make the distinction between "global cos Ive never seen it before" vs "global and Ive seen an assignment". Does any of that make sense? Mark. From mhammond@skippinet.com.au Mon Feb 21 22:44:33 2000 From: mhammond@skippinet.com.au (Mark Hammond) Date: Tue, 22 Feb 2000 09:44:33 +1100 Subject: [Compiler-sig] FYI: python/nondist/src/Compiler In-Reply-To: <14513.26116.110383.957967@goon.cnri.reston.va.us> Message-ID: [Jeremy writes] > Do tell! What is this other project? If I told you, I would have to kill you! :-) > MH> So, I had a think about reusing some of this framework for other > MH> compilers, and come to the conclusion that not as much as I had > MH> hoped can be shared ( But still enough that it makes it > MH> worthwhile. I can see the P2C directory stuff being shared, > MH> along with the basic tree stuff you have. > > It depends what the next steps for any of these compilers are. The > code I have checked in is pretty primitive; there isn't much to share, > just because there isn't much there at all. The "problem" I saw was simply that your code is very tied to Python byte-code - which is what you would expect for a byte-code compiler :-) Your code shares quite a bit of logic with (say) P2C, but it is sprinkled in the middle of the emit() calls - ie, while the general logic is the same, I couldnt see an obvious way to remove the knowledge of the specific interpreter/vm you are targetting. > I hope to refactor the pyassem module and revise the code generator a > bit. Instead of emitting straight-line code, with refs -- misnamed > StackRefs -- to other locations in the instruction stream, I would > like to generate code blocks that contain references to other blocks. > This intermediate representation would be more amenable to analysis > and still easy to generate bytecode from. What is your intention with this compiler? To be an emulation of the existing compiler or to become an (optionally) optimizing compiler? > I had hoped to do some dataflow analysis, but won't have time to work > on it in the near future. It's also possible that Jon Riehl will > release his dataflow code soon, which might make sense to incorporate > as part of the standard tool suite. Cool! What I personally have a need for is some sort of "name/type tracking" - at this stage limited to simple assignments within a scoped block. Eg: class C: pass c = C() d = c I would dearly like a system that could tell me that c and d are instances of the class C. But I also recognize how quickly that becomes a general type inferencing problem, and I dont have the time to get sidetracked by that right now (especially when a few very smart people have already stated their intention :-) So Im just punting that for now :-) Mark. From billtut@microsoft.com Tue Feb 22 17:25:02 2000 From: billtut@microsoft.com (Bill Tutt) Date: Tue, 22 Feb 2000 09:25:02 -0800 Subject: [Compiler-sig] RE: Compiler-sig digest, Vol 1 #7 - 2 msgs Message-ID: <4D0A23B3F74DD111ACCD00805F31D8101D8BCD5A@RED-MSG-50> > From: Jeremy Hylton > MH> If you just change the COPYRIGHT file to reflect that it was > MH> taken from the P2C project and renamed to "compiler.tools" > MH> everyone should be happy I would think... > > Sounds good to me; makes sense to move the visitor code to > compiler.tools. This change okay with you, Greg & Bill? > Works for me. Bill From gstein@lyra.org Thu Feb 24 22:38:38 2000 From: gstein@lyra.org (Greg Stein) Date: Thu, 24 Feb 2000 14:38:38 -0800 (PST) Subject: [Compiler-sig] P2C stuff (was: Compiler-sig digest, Vol 1 #7 - 2 msgs) In-Reply-To: <4D0A23B3F74DD111ACCD00805F31D8101D8BCD5A@RED-MSG-50> Message-ID: On Tue, 22 Feb 2000, Bill Tutt wrote: > > From: Jeremy Hylton > > MH> If you just change the COPYRIGHT file to reflect that it was > > MH> taken from the P2C project and renamed to "compiler.tools" > > MH> everyone should be happy I would think... > > > > Sounds good to me; makes sense to move the visitor code to > > compiler.tools. This change okay with you, Greg & Bill? > > > > Works for me. I'm a bit unclear on what is proposed here... If (portions of?) P2C becomes compiler.tools, then I don't think that you'd want to put more stuff in there (implying that it was Bill's/mine). Now, if you said "files X,Y,Z are subject to the following copyright statement and license: ...", then I'd be just as happy. Or if the P2C stuff was placed into "compiler.p2c". Talking with Guido at the '96 IPC, he described his attitude with Python. I'll take the same position with P2C: "Do what you want with it, but please don't try to pass it off as your own work." That said: feel free to organize the stuff however you feel, and don't feel obligated to run your decision past me. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond@skippinet.com.au Thu Feb 24 23:08:22 2000 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 25 Feb 2000 10:08:22 +1100 Subject: [Compiler-sig] P2C stuff (was: Compiler-sig digest, Vol 1 #7 - 2 msgs) In-Reply-To: Message-ID: > > > MH> If you just change the COPYRIGHT file to reflect that it was > > > MH> taken from the P2C project and renamed to "compiler.tools" > > > MH> everyone should be happy I would think... > > I'm a bit unclear on what is proposed here... If (portions of?) P2C > becomes compiler.tools, then I don't think that you'd want to put more > stuff in there (implying that it was Bill's/mine). Well, to be honest, I _would_ be happy with that :-) > Now, if you said "files X,Y,Z are subject to the following copyright > statement and license: ...", then I'd be just as happy. Of course. Whatever. If there ended up 1 or 2 extra files, then their copyright could just be asserted in the source code. If there end up being lots of stuff, then the copyright could be changed to reflect the files taken from the P2C project, and those new files. People around here understand how this stuff works. > Or if the P2C stuff was placed into "compiler.p2c". It doesnt make as much sense to have "compiler.p2c.tranformer". compiler.p2c is where the p2c specific code would live. > "Do what you want with it, but please don't try to pass it off as your > own work." Geez - I hope you didnt seriously think that anyone here was trying to, or had any intention of doing this? Your email starts with the quote about how it is important to retain the copyright. Just because I didnt "dot every i and cross every t" but assumed that the readers are capable of knowing how the copyright stuff should work, there is no need to start throwing stuff like this around :-( Mark. > > That said: feel free to organize the stuff however you feel, and don't > feel obligated to run your decision past me. > > Cheers, > -g > > -- > Greg Stein, http://www.lyra.org/ > > > _______________________________________________ > Compiler-sig mailing list > Compiler-sig@python.org > http://www.python.org/mailman/listinfo/compiler-sig > From gstein@lyra.org Thu Feb 24 23:30:46 2000 From: gstein@lyra.org (Greg Stein) Date: Thu, 24 Feb 2000 15:30:46 -0800 (PST) Subject: [Compiler-sig] copyright/license BS (was: P2C stuff) In-Reply-To: Message-ID: On Fri, 25 Feb 2000, Mark Hammond wrote: >... > > "Do what you want with it, but please don't try to pass it off as your > > own work." > > Geez - I hope you didnt seriously think that anyone here was trying to, or > had any intention of doing this? Your email starts with the quote about how > it is important to retain the copyright. Just because I didnt "dot every i > and cross every t" but assumed that the readers are capable of knowing how > the copyright stuff should work, there is no need to start throwing stuff > like this around :-( No... this is saying "do whatever. I don't care." In no way do I believe anybody *is* trying to claim ownership. I'm simply saying that Jeremy (and/or whoever) can do what they want. Do whatever. No need to check with me. Heck... many of the modules that I've written, I call Public Domain. In other words: I'm not even asserting a copyright! The reason why I label stuff public domain? Because I don't want to deal with mail threads exactly like this one. Copyright this, copyright that. Did you send a disclaimer? How about a wet signature form? Did you make sure to label each piece properly? Is that yours, or theirs? Do these licenses work together? Is one viral? blah fucking blah. Over the past couple years, it seems that people are overly touchy on the whole damn licensing/copyright bullshit. I don't like it, if you can't tell. I wish people would be more cooperative and less worried about this stuff. Years ago, Mark and I wrote the pythoncom stuff. It never had a license or even much of a copyright assertion -- it just wasn't a big deal. Last week, Mark finally applied one because today's environment demands it. I wrote imputil last year and published it. About six months ago, I got a query "what's the license on it?" I never had bothered because I'd rather just write useful code; I replied "public domain". Regarding P2C: Take my code. Do what you want. Leave some credit in there for me. 'nuf said, -g -- Greg Stein, http://www.lyra.org/ From tim_one@email.msn.com Fri Feb 25 07:32:10 2000 From: tim_one@email.msn.com (Tim Peters) Date: Fri, 25 Feb 2000 02:32:10 -0500 Subject: [Compiler-sig] copyright/license BS (was: P2C stuff) In-Reply-To: Message-ID: <001501bf7f62$6d0aa540$b62d153f@tim> [Greg Stein] > ... > Heck... many of the modules that I've written, I call Public Domain. In > other words: I'm not even asserting a copyright! > > The reason why I label stuff public domain? Because I don't want to deal > with mail threads exactly like this one. Copyright this, copyright that. > Did you send a disclaimer? How about a wet signature form? Did you make > sure to label each piece properly? Is that yours, or theirs? Do these > licenses work together? Is one viral? blah fucking blah. For anyone able to, I highly recommend public domain too! In fact, the only thing of mine I *didn't* put in the public domain was the Emacs pymode, because I inherited a skeleton mode that already had a stinkin' copyright notice so felt I had to follow suit. This bad decision dogged me for 5+ years, hassling over & over to get the copyright assigned to the FSF. At one point I filed paperwork to put pymode in the public domain so FSF could claim it was always theirs -- but that didn't fly either. It looks like the FSF does have the copyright now, as just last week I finally got back a copy of the latest batch of papers I signed.. Note that the Icon language has been in the public domain from the start -- people could learn a lot from Ralph Griswold . twaddle-aversive-ly y'rs - tim