Indentation oddness...
Consider the code: code = "def Foo():\n\n pass\n\n " This code is malformed in that the final indentation (2 spaces) does not agree with the previous indentation of the pass statement (4 spaces). Or maybe it's just fine if you take the blank lines should be ignored statement from the docs to be true. So let's look at different ways I can consume this code. If I use compile to compile this: compile(code, 'foo', 'single') I get an IndentationError: unindent does not match any outer indentation level But if I put this in a file: f= file('indenttest.py', 'w') f.write(code) f.close() import indenttest It imports just fine. If I run it through the tokenize module it also tokenizes just fine:
import tokenize from cStringIO import StringIO tokenize.tokenize(StringIO(code).readline) 1,0-1,3: NAME 'def' 1,5-1,8: NAME 'Foo' 1,8-1,9: OP '(' 1,9-1,10: OP ')' 1,10-1,11: OP ':' 1,11-1,12: NEWLINE '\n' 2,0-2,1: NL '\n' 3,0-3,4: INDENT ' ' 3,4-3,8: NAME 'pass' 3,8-3,9: NEWLINE '\n' 4,0-4,1: NL '\n' 5,0-5,0: DEDENT '' 5,0-5,0: ENDMARKER ''
And if it fails anywhere it would seem tokenization is where it should fail - especially given that tokenize.py seems to report this error on other occasions. And stranger still if I add a new line then it will even compile fine: compile(code + '\n', 'foo', 'single') Which seems strange because in either case all of the trailing lines are blank lines and as such should basically be ignored according to the documentation. Is there some strange reason why compile rejects what everything else agrees is perfectly valid code?
On 2009-05-29 19:08, Dino Viehland wrote:
Consider the code:
code = "def Foo():\n\n pass\n\n "
This code is malformed in that the final indentation (2 spaces) does not agree with the previous indentation of the pass statement (4 spaces). Or maybe it's just fine if you take the blank lines should be ignored statement from the docs to be true. So let's look at different ways I can consume this code.
If I use compile to compile this:
compile(code, 'foo', 'single')
I get an IndentationError: unindent does not match any outer indentation level
But if I put this in a file:
f= file('indenttest.py', 'w') f.write(code) f.close() import indenttest
It imports just fine.
The 'single' mode, which is used for the REPL, is a bit different than 'exec', which is used for modules. This difference lets you insert "blank" lines of whitespace into a function definition without exiting the definition. Ending with a truly empty line does not cause the IndentationError, so the REPL can successfully compile the code, signaling that the user has finished typing the function. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
The 'single' mode, which is used for the REPL, is a bit different than 'exec', which is used for modules. This difference lets you insert "blank" lines of whitespace into a function definition without exiting the definition. Ending with a truly empty line does not cause the IndentationError, so the REPL can successfully compile the code, signaling that the user has finished typing the function.
Sorry, I probably should have mentioned this but it repros w/ compile(..., "exec") as well:
code = "def Foo():\n\n pass\n\n " compile(code, 'foo', 'exec') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "foo", line 5
IndentationError: unindent does not match any outer indentation level It also repros when passing in PyCF_DONT_IMPLY_DEDENT for flags under single and exec.
I usually append some extra newlines before passing a string to compile(). That's the usual work-around. There's probably a subtle bug in the tokenizer when reading from a string -- if you find it, please upload a patch to the tracker! --Guido On Fri, May 29, 2009 at 5:52 PM, Dino Viehland <dinov@microsoft.com> wrote:
The 'single' mode, which is used for the REPL, is a bit different than 'exec', which is used for modules. This difference lets you insert "blank" lines of whitespace into a function definition without exiting the definition. Ending with a truly empty line does not cause the IndentationError, so the REPL can successfully compile the code, signaling that the user has finished typing the function.
Sorry, I probably should have mentioned this but it repros w/ compile(..., "exec") as well:
code = "def Foo():\n\n pass\n\n " compile(code, 'foo', 'exec') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "foo", line 5
IndentationError: unindent does not match any outer indentation level
It also repros when passing in PyCF_DONT_IMPLY_DEDENT for flags under single and exec. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Unfortunately my problem is the opposite one - trying to emulate what compile does for IronPython rather than just trying to make some code compile. So adding newlines doesn't help me. But this case isn't really that important - it was just a wacky corner case I ran into while trying to get other behavior right. I think I can safely ignore this one especially if it's just a bug.
-----Original Message----- From: gvanrossum@gmail.com [mailto:gvanrossum@gmail.com] On Behalf Of Guido van Rossum Sent: Friday, May 29, 2009 7:20 PM To: Dino Viehland Cc: Robert Kern; python-dev@python.org Subject: Re: [Python-Dev] Indentation oddness...
I usually append some extra newlines before passing a string to compile(). That's the usual work-around. There's probably a subtle bug in the tokenizer when reading from a string -- if you find it, please upload a patch to the tracker!
--Guido
The 'single' mode, which is used for the REPL, is a bit different
On Fri, May 29, 2009 at 5:52 PM, Dino Viehland <dinov@microsoft.com> wrote: than
'exec', which is used for modules. This difference lets you insert "blank" lines of whitespace into a function definition without exiting the definition. Ending with a truly empty line does not cause the IndentationError, so the REPL can successfully compile the code, signaling that the user has finished typing the function.
Sorry, I probably should have mentioned this but it repros w/ compile(..., "exec") as well:
code = "def Foo():\n\n pass\n\n " compile(code, 'foo', 'exec') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "foo", line 5
IndentationError: unindent does not match any outer indentation level
It also repros when passing in PyCF_DONT_IMPLY_DEDENT for flags under single and exec. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python- dev/guido%40python.org
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Robert Kern wrote:
The 'single' mode, which is used for the REPL, is a bit different than 'exec', which is used for modules. This difference lets you insert "blank" lines of whitespace into a function definition without exiting the definition.
All that means is that the REPL needs to keep reading lines until it gets a completely blank one. I don't see why the compiler has to treat the source any differently once the REPL has decided how much text to feed it. -- Greg
On 2009-05-30 21:02, Greg Ewing wrote:
Robert Kern wrote:
The 'single' mode, which is used for the REPL, is a bit different than 'exec', which is used for modules. This difference lets you insert "blank" lines of whitespace into a function definition without exiting the definition.
All that means is that the REPL needs to keep reading lines until it gets a completely blank one. I don't see why the compiler has to treat the source any differently once the REPL has decided how much text to feed it.
Not true. The REPL will keep reading lines until compile(code,'<string>','single') passes without a SyntaxError. You can put in blank lines when line continuation is implicit, like in the middle of a list. This is the reason that there is a 'single' mode in the first place, to determine when you've stopped typing. It's easier to add the grammar rule that a block does not end with a line of whitespace to the compiler than to implement all of the context-specific special cases for pure empty lines outside of the compiler. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
participants (4)
-
Dino Viehland -
Greg Ewing -
Guido van Rossum -
Robert Kern