[Python-bugs-list] [ python-Bugs-501622 ] compile() 'exec' mode SyntaxError

[noreply@sourceforge.net]

Bugs item #501622, was opened at 2002-01-10 04:01
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=501622&group_id=5470

Category: Parser/Compiler
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: David Bolen (db3l)
Assigned to: Nobody/Anonymous (nobody)
Summary: compile() 'exec' mode SyntaxError

Initial Comment:
If you have a module that you wish to compile using 
the builtin compile() function (in 'exec' mode), it 
will fail with a SyntaxError if that module does not 
have a newline as its final token.

The same module can be executed directly by the 
interpreter, or imported by another module, and Python 
can properly compile and save a pyc for the module.

I believe the difference is rooted in the fact that 
the tokenizer (tokenizer.c, in tok_nextc()) 
will "fake" a newline at the end of a file if it 
doesn't find one, but it will not do so when 
tokenizing a string buffer.

What I'm not certain of is whether faking such a token 
for strings as well won't break something else (such 
as when parsing a string for an expression rather than 
a full module).  But without such a change, you have a 
state where a module that works (and compiles) in 
other circumstances cannot be read into memory and 
compiled with the compile() builtin.

This came up while tracking down a problem with 
failures using Gordan McMillan's Installer package 
which compiles modules using compile() before 
including them in the archive.

I believe this is true for all releases since at least 
1.5.2.

-- David

----------------------------------------------------------------------

>Comment By: Neil Schemenauer (nascheme)
Date: 2002-03-22 22:07

Message:
Logged In: YES 
user_id=35752

I ran into this bug myself when writing the PTL compiler.
Here's a test case:

code = "def foo():\n  pass"
open("bug.py", "w").write(code)
import bug # works
compile(code, "<string>", "exec") # doesn't work

I traced this bug to tok_nextc.  If the input is coming from
a file and the last bit of input doesn't end with a newline
then one is faked.  This doesn't happen if the input is
coming from a string.  I spent time trying to figure out
how to fix it but the tok_nextc code is hairy and whatever
I tried broke something else.

----------------------------------------------------------------------

Comment By: Neil Schemenauer (nascheme)
Date: 2002-03-22 22:07

Message:
Logged In: YES 
user_id=35752

I ran into this bug myself when writing the PTL compiler.
Here's a test case:

code = "def foo():\n  pass"
open("bug.py", "w").write(code)
import bug # works
compile(code, "<string>", "exec") # doesn't work

I traced this bug to tok_nextc.  If the input is coming from
a file and the last bit of input doesn't end with a newline
then one is faked.  This doesn't happen if the input is
coming from a string.  I spent time trying to figure out
how to fix it but the tok_nextc code is hairy and whatever
I tried broke something else.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=501622&group_id=5470