[Patches] [ python-Patches-526840 ] PEP 263 Implementation

[noreply@sourceforge.net]

Patches item #526840, was opened at 2002-03-07 09:55
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=526840&group_id=5470

Category: Parser/Compiler
Group: Python 2.3
>Status: Closed
>Resolution: Out of Date
Priority: 7
Submitted By: Martin v. Löwis (loewis)
Assigned to: M.-A. Lemburg (lemburg)
Summary: PEP 263 Implementation

Initial Comment:
The attached patch implements PEP 263. The following
differences to the PEP (rev. 1.8) are known:

- The implementation interprets "ASCII compatible" as
meaning "bytes below 128 always denote ASCII
characters", although this property is only used for
",', and \. There have been other readings of "ASCII
compatible", so this should probably be elaborated in
the PEP.

- The check whether all bytes follow the declared or
system encoding (including comments and string
literals) is only performed if the encoding is "ascii".


----------------------------------------------------------------------

>Comment By: Martin v. Löwis (loewis)
Date: 2002-08-04 18:59

Message:
Logged In: YES 
user_id=21627

This patch has been superceded by 534304.

----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg)
Date: 2002-04-11 18:23

Message:
Logged In: YES 
user_id=38388

Apart from the codec changes, the patch looks ok.

I would still like two APIs for the two different codec tasks, 
though. I don't expect anything much to change in the
codecs, so maintenance is not an issue.


----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-03-21 11:25

Message:
Logged In: YES 
user_id=21627

Version 2 of this patch implements revision 1.11 of the PEP
(phase 1). The check of the complete source file for
compliance with the declared encoding is implemented by
decoding the input line-by-line; I believe that for all
supported encodings, this is not different compared to
decoding the entire source file at once.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-03-07 19:24

Message:
Logged In: YES 
user_id=21627

Changing the decoding functions will not result in one
additional function, but in two of them: you'll also get
PyUnicode_DecodeRawUnicodeEscapeFromUnicode.

That seems quite unmaintainable to me: any change now needs
to propagate into four functions. OTOH, I don't think that
the code that allows parsing a variable-sized strings is
overly complicated.

----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg)
Date: 2002-03-07 19:01

Message:
Logged In: YES 
user_id=38388

Ok, I've had a look at the patch. 

It looks good except for the overly 
complicated implementation of the 
unicode-escape codec. 

Even though there's a bit of code duplication, 
I'd prefer to have two separate functions here: 
one for the standard char* pointer type and 
another one for Py_UNICODE*, ie.

PyUnicode_DecodeUnicodeEscape(char*...)
and
PyUnicode_DecodeUnicodeEscapeFromUnicode(Py_UNICODE*...)

This is easier to support and gives better
performance since the compiler can optimize
the two functions making different 
assumptions.

You'll also need to include a name mangling
at the top of the header for the new API.


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-03-07 15:06

Message:
Logged In: YES 
user_id=6380

I've set the group to Python 2.3 so the priority has some
context (I'd rather you move the priority down to 5 but I
understand this is your personal priority).

I haven't accepted the PEP yet (although I expect I will),
so please don't check this in yet (if you feel it needs to
be saved in CVS, use a branch).


----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg)
Date: 2002-03-07 12:06

Message:
Logged In: YES 
user_id=38388

Thank you !

I'll add a note to the PEP about the way the first two lines
are processed (removing the ASCII mention...).

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-03-07 10:11

Message:
Logged In: YES 
user_id=21627

A note on the implementation strategy: it turned out that
communicating the encoding into the abstract syntax was the
biggest challenge. 

To solve this, I introduced encoding_decl pseudo node: it is
an unused non-terminal whose STR() is the encoding, and
whose only child is the true root of the syntax tree. As
such, it is the only non-terminal which has a STR value.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=526840&group_id=5470