[Python-bugs-list] [ python-Bugs-501622 ] compile() 'exec' mode SyntaxError
noreply@sourceforge.net
noreply@sourceforge.net
Fri, 22 Mar 2002 14:20:33 -0800
Bugs item #501622, was opened at 2002-01-09 23:01
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=501622&group_id=5470
Category: Parser/Compiler
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: David Bolen (db3l)
Assigned to: Nobody/Anonymous (nobody)
Summary: compile() 'exec' mode SyntaxError
Initial Comment:
If you have a module that you wish to compile using
the builtin compile() function (in 'exec' mode), it
will fail with a SyntaxError if that module does not
have a newline as its final token.
The same module can be executed directly by the
interpreter, or imported by another module, and Python
can properly compile and save a pyc for the module.
I believe the difference is rooted in the fact that
the tokenizer (tokenizer.c, in tok_nextc())
will "fake" a newline at the end of a file if it
doesn't find one, but it will not do so when
tokenizing a string buffer.
What I'm not certain of is whether faking such a token
for strings as well won't break something else (such
as when parsing a string for an expression rather than
a full module). But without such a change, you have a
state where a module that works (and compiles) in
other circumstances cannot be read into memory and
compiled with the compile() builtin.
This came up while tracking down a problem with
failures using Gordan McMillan's Installer package
which compiles modules using compile() before
including them in the archive.
I believe this is true for all releases since at least
1.5.2.
-- David
----------------------------------------------------------------------
>Comment By: Guido van Rossum (gvanrossum)
Date: 2002-03-22 17:20
Message:
Logged In: YES
user_id=6380
> the tok_nextc code is hairy and whatever
> I tried broke something else.
That's exactly what happened to me when I tried to fix this
myself long ago. :-(
The workaround is simple enough: whoever calls compile()
should append a newline to the string just to be sure.
----------------------------------------------------------------------
Comment By: Neil Schemenauer (nascheme)
Date: 2002-03-22 17:07
Message:
Logged In: YES
user_id=35752
I ran into this bug myself when writing the PTL compiler.
Here's a test case:
code = "def foo():\n pass"
open("bug.py", "w").write(code)
import bug # works
compile(code, "<string>", "exec") # doesn't work
I traced this bug to tok_nextc. If the input is coming from
a file and the last bit of input doesn't end with a newline
then one is faked. This doesn't happen if the input is
coming from a string. I spent time trying to figure out
how to fix it but the tok_nextc code is hairy and whatever
I tried broke something else.
----------------------------------------------------------------------
Comment By: Neil Schemenauer (nascheme)
Date: 2002-03-22 17:07
Message:
Logged In: YES
user_id=35752
I ran into this bug myself when writing the PTL compiler.
Here's a test case:
code = "def foo():\n pass"
open("bug.py", "w").write(code)
import bug # works
compile(code, "<string>", "exec") # doesn't work
I traced this bug to tok_nextc. If the input is coming from
a file and the last bit of input doesn't end with a newline
then one is faked. This doesn't happen if the input is
coming from a string. I spent time trying to figure out
how to fix it but the tok_nextc code is hairy and whatever
I tried broke something else.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=501622&group_id=5470