[ python-Bugs-780730 ] source files using encoding ./. universal newlines
SourceForge.net
noreply at sourceforge.net
Tue Feb 21 10:45:42 CET 2006
Bugs item #780730, was opened at 2003-07-31 11:16
Message generated for change (Comment added) made by lemburg
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=780730&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Parser/Compiler
Group: Python 2.5
>Status: Closed
Resolution: Fixed
Priority: 5
Submitted By: atsuo ishimoto (ishimoto)
Assigned to: M.-A. Lemburg (lemburg)
Summary: source files using encoding ./. universal newlines
Initial Comment:
Universal Newline Support doesn't work for source files
that contain encoding definition.
----------------------------------------------------------------------
>Comment By: M.-A. Lemburg (lemburg)
Date: 2006-02-21 10:45
Message:
Logged In: YES
user_id=38388
Closing the bug. Thanks Walter.
----------------------------------------------------------------------
Comment By: Walter Dörwald (doerwalter)
Date: 2006-02-20 23:06
Message:
Logged In: YES
user_id=89016
It looks to me that way.
Any comments from the OP?
----------------------------------------------------------------------
Comment By: Georg Brandl (gbrandl)
Date: 2006-02-20 22:48
Message:
Logged In: YES
user_id=849994
So this is resolved now?
----------------------------------------------------------------------
Comment By: Walter Dörwald (doerwalter)
Date: 2006-02-20 22:42
Message:
Logged In: YES
user_id=89016
The changes to the codecs done in Python 2.4 added support
for universal newlines:
Python 2.4.1 (#2, Mar 31 2005, 00:05:10)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin
Type "help", "copyright", "credits" or "license" for more
information.
>>> open("foo.py", "wb").write("# -*- coding: iso-8859-1
-*-\rprint 17\rprint 23\r")
>>> import foo
17
23
>>>
----------------------------------------------------------------------
Comment By: Neal Norwitz (nnorwitz)
Date: 2005-10-02 07:50
Message:
Logged In: YES
user_id=33168
I don't see a clear resolution here. Is there something we
can/should do to fix this problem in 2.5?
----------------------------------------------------------------------
Comment By: Jack Jansen (jackjansen)
Date: 2003-08-05 12:48
Message:
Logged In: YES
user_id=45365
You misunderstand what I tried to say (or I mis-said it:-): there is
no such thing as mode "rbU", check the code in fileobject.c.
There is "r" == "rt" for text mode, "rb" for binary mode,
"U"=="rU" for universal newline textmode. With "rU" the
underlying file is opened in binary mode, so I don't think we'll
have the control-Z problem.
----------------------------------------------------------------------
Comment By: M.-A. Lemburg (lemburg)
Date: 2003-08-05 09:31
Message:
Logged In: YES
user_id=38388
Thanks John.
Not sure whether any of codecs would actually use 0x1A,
but using "rbU" sounds like the safer approach then.
----------------------------------------------------------------------
Comment By: John J Smith (johnjsmith)
Date: 2003-08-04 21:34
Message:
Logged In: YES
user_id=830565
In MS Windows, a '\x1a' (Ctrl-Z) in a file will be treated
as EOF, unless the file is opened with 'rb'.
----------------------------------------------------------------------
Comment By: M.-A. Lemburg (lemburg)
Date: 2003-08-04 17:57
Message:
Logged In: YES
user_id=38388
Jack, I was just looking at the code I posted and the one
in fileobect.c. The latter enables universal newline support
whenever it sees a 'U' in the mode string, so I throught that
adding a 'U' to the mode would be enough.
The only system where 'b' does make a difference that I'm
aware of is Windows, so you may want to check whether it
makes a difference there.
----------------------------------------------------------------------
Comment By: Jack Jansen (jackjansen)
Date: 2003-08-04 10:24
Message:
Logged In: YES
user_id=45365
There's no such things as "rbU", I think, but simply "rU" should
work. As far as I know the only difference between "r" and "rb" is
newline conversion, right? If there are C libraries that do more
then we should implement "rbU".
About 2.3.1 compatibility: technically we could break workarounds
people have done themselves, but I think the chances are slim.
----------------------------------------------------------------------
Comment By: M.-A. Lemburg (lemburg)
Date: 2003-08-04 10:21
Message:
Logged In: YES
user_id=38388
Uhm, we don't impose Universal readline support on the codecs.
They would just get a stream object that happens to know
about universal newlines and work with it. That's completely
in line with the codec spec.
I'm +1 on adding this to 2.3.1.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2003-08-03 22:22
Message:
Logged In: YES
user_id=21627
That would work indeed. The question is whether we can
impose support for universal newlines on all codecs "out
there", for Python 2.3.1, when 2.3 makes no such requirement.
----------------------------------------------------------------------
Comment By: M.-A. Lemburg (lemburg)
Date: 2003-08-03 21:31
Message:
Logged In: YES
user_id=38388
I'm not sure this is correct: unless the codecs implement
their own .readline() implementation, the one in codecs.py
is used and that simply delegates the readline request to
the underlying stream object.
Now. since the stream object in the source code reader is
a plain Python file object, currently opened in "rb" mode,
changing the mode to "rbU" should be enough to get
universal readline support for all such codecs.
The relevant code is in Parser/tokenizer.c:fp_setreadl():
static int
fp_setreadl(struct tok_state *tok, const char* enc)
{
PyObject *reader, *stream, *readline;
/* XXX: constify filename argument. */
stream = PyFile_FromFile(tok->fp, (char*)tok->filename,
"rb", NULL);
if (stream == NULL)
return 0;
reader = PyCodec_StreamReader(enc, stream, NULL);
Py_DECREF(stream);
if (reader == NULL)
return 0;
readline = PyObject_GetAttrString(reader, "readline");
Py_DECREF(reader);
if (readline == NULL)
return 0;
tok->decoding_readline = readline;
return 1;
}
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2003-08-03 21:19
Message:
Logged In: YES
user_id=21627
The only way to implement that would be to add 'U' support
for all codecs, and open the codec with the 'U' flag. As
this will also affect codecs not part of Python, we cannot
fix this bug in Python 2.3, but have to defer a solution to
Python 2.4.
----------------------------------------------------------------------
Comment By: Jack Jansen (jackjansen)
Date: 2003-08-01 11:45
Message:
Logged In: YES
user_id=45365
The submitter isn't calling open explicitly, but PEP278 is also about
the implicit opening of sourcefiles by the interpreter.
My guess (but I don't know the PEP263 implementation at all) is
that when you specify an encoding the normal code that the
interpreter uses to read sourcefiles is bypassed, and replaced by
something that does the encoding magic. Apparently this code
does not do universal newline conversion, which it should.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2003-08-01 10:22
Message:
Logged In: YES
user_id=21627
I see. This is not about PEP 278, though, as you are not
calling any open() function, and passing no 'U' argument to
it - cross-platform newlines should work independent of that
PEP.
----------------------------------------------------------------------
Comment By: atsuo ishimoto (ishimoto)
Date: 2003-08-01 09:35
Message:
Logged In: YES
user_id=463672
Attached file has encoding definition and it's newline is '\r'.
Executing this script on Windows 2000, I got following error.
But it runs fine if I remove encoding definition.
C:\Python23>python .\test.py
File ".\test.py", line 2
a() print 'hi'
^
SyntaxError: invalid syntax
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2003-08-01 09:04
Message:
Logged In: YES
user_id=21627
It's not clear to me what the problem is: Source code files
and universal newline support have nothing to do with each
other.
Can you attach a small example that demonstrates the problem?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=780730&group_id=5470
More information about the Python-bugs-list
mailing list