[IronPython] Encoding problem

[Dino Viehland]

ASCII is actually a fine value here it's just that we don't currently support the # -*- encoding syntax.  What we do currently support is the normal Unicode headers at the beginning of files that Windows typically does (CPython also will recognize these).  If that's present then we'll decode the file as Unicode otherwise we'll decode it as ASCII (this is a feature of the BCL that we pick up for free).

We do have an active bug to support PEP-263 file encoding syntax - which is what you're using here.  Based upon your feedback I've raised the priority of this bug but I'm still not sure it'll make our next release.  What we'd do then is get the encoding and then re-open the file w/ the correct encoding.  

It seems like you have a reasonable work around for yourself for the time being but I don't think this will be a portable solution that will work from machine to machine - for example if the default encoding is some non-Unicode encoding then many characters won't be available.

-----Original Message-----
From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Michael Tan
Sent: Wednesday, January 04, 2006 6:56 AM
To: users at lists.ironpython.com
Subject: [IronPython] Encoding problem

Dear All,
    Here I am. :) This time, I have another silly problem about encoding of 
IronPython. According to my experience in programming, now IP can support 
unicode under interactive mode. If I put the unicode into .py file, the 
unicode code can not be displayed correctly.And I followed the Python 
Specification and tried to add the encoding declaration into .py file like 
this:
     # -*- coding: utf-8 -*-
But this declaration doesn't work.
    I found that what results in the encoding problem. The fromFile method 
in Parser.cs is the key factor. Here is the source of fromFile method:
     public static Parser fromFile(CompilerContext context, string 
filename) {
            string data;
            // we choose ASCII by default, if the file has a Unicode header 
though
            // we'll automatically get it as unicode.
            StreamReader sr = new StreamReader(File.OpenRead(filename), 
System.Text.Encoding.ASCII);
            using (sr) {
                data = sr.ReadToEnd();
                sr.Close();
            }
            return new Parser(context, filename, new Tokenizer(context, 
data.ToCharArray()));
        }
     I revised the System.Text.Encoding.ASCII to 
System.Text.Encoding.Default. The encoding problem was solved. I do hope 
receive the official comment about this revision. Thank you very much.

Regards,
Michael