Mailman 3 January 2002 - Python-Dev

Unicode support in getargs.c
by Jack Jansen Jan. 6, 2002

Jan. 6, 2002

I posted a question on Unicode support in getargs.c last month (working on a different project), but now that I'm trying to support unicode-based APIs more seriously I find that it leaves even more to be desired. I'd like to help to fix this, but I need some direction on how things should be fixed. Here are some of the issues I ran in today: - Unicode objects have a companion string object, meaning that you can pass a unicode object to an "s" format and have the right thing happen. String objects have no such accompanying unicode object, and I think they should have. Right now you cannot pass a string object when the C routine expects a unicode object. - There is no unicode equivalent of "c", the single character. - "u#" does something useful, but something completely different from what "s#" does. More to the point, it probably does something dangerous, if I understand correctly. If I write a C routine with an "u#" format and the Python code passes a string object the string object will be used as a buffer object and its binary contents will be interpreted as unicode. If the argument in question is a filename this will produce very surprising results:-) I'd like unicode objects to be get a little more first class citizenship, especially in the light of operating systems that are primarily (or exclusively) unicode based, such as Mac OS X or Windows CE, to sum things up.

5 17

Unicode strings as filenames
by Skip Montanaro Jan. 6, 2002

Jan. 6, 2002

What's the correct way to deal with filenames in a Unicode environment? Consider this: >>> import site >>> site.encoding 'latin-1' >>> a = "abc\xe4\xfc\xdf.txt" >>> u = unicode (a, "latin-1") >>> uu = u.encode ("utf-8") >>> open(a, "w") <open file 'abcäüß.txt', mode 'w' at 0x823c2a0> >>> open(u, "w") <open file 'abcäüß.txt', mode 'w' at 0x823a1e8> >>> open(uu, "w") <open file 'abcÃ¤Ã¼Ã.txt', mode 'w' at 0x81d6160> If I change my site's default encoding back to ascii, the second open fails: >>> import site >>> site.encoding 'ascii' >>> a = "abc\xe4\xfc\xdf.txt" >>> u = unicode (a, "latin-1") >>> uu = u.encode ("utf-8") >>> open(a, "w") <open file 'abcäüß.txt', mode 'w' at 0x822b448> >>> open(u, "w") Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeError: ASCII encoding error: ordinal not in range(128) >>> open(uu, "w") <open file 'abcÃ¤Ã¼Ã.txt', mode 'w' at 0x822d260> as I expect it should. The third open is a problem as well, even though it succeeds with either encoding. (Why doesn't it fail when the default encoding is ascii?) My thought is that before using a plain string or a unicode string as a filename it should first be coerced to a unicode string with the default encoding, something like: if type(fname) == types.StringType: fname = unicode(fname, site.encoding) elif type(fname) == types.UnicodeType: fname = fname.encode(site.encoding) else: raise TypeError, ("unrecognized type for filename: %s"%type(fname)) Is that the correct approach? Apparently Python's file object doesn't do this under the covers. Should it? Thx, Skip

4 24

Re: [XML-SIG] printing Unicode xml to StringIO
by Martin v. Loewis Jan. 6, 2002

Jan. 6, 2002

[Over to python-dev. Jaco noticed that writing Unicode objects to a StringIO object stopped working in 2.2, see http://mail.python.org/pipermail/xml-sig/2001-December/006891.html ] Marc-Andre writes > Actually, I think that this is a bug in the documentation, not the > code. StringIO and cStringIO were never meant to work on anything but > strings and memory buffers. IMO, "strings" should include both byte strings and Unicode strings. Mixing them may not be allowed, but that is a different story. In fact, there is an open bug (#216388) that cStringIO rejects Unicode objects. If that gets fixed, we get the funny scenario that StringIO rejects Unicode object, whereas cStringIO accepts them. > The note that Fred added to the docs about StringIO's capability of > storing Unicode in it's buffer list is simply an artifact of the > implementation. There are many developers who take this note literally. Claiming that this was not intentional is a mistake. > Please use the .encode() method on Unicode objects before writing > them to a StringIO object. If you want to end up with a byte string, this is a good idea. But I think it is pointless to require encoding them when you want to end up with a Unicode string; you'd have to invoke unicode() on the result, for no apparent reason but a bug in the StringIO implementation. Regards, Martin

3 8

Re: [Python-Dev] RPM *.spec file
by Sean Reifschneider Jan. 6, 2002

Jan. 6, 2002

>> Could someone point me to the RPM *.spec file with which was built the >> Python 2.2 (ftp://ftp.python.org/pub/python/2.2/rpms/)? The nice thing about an source RPM file (.src.rpm) is that includes *ALL* things necessary to reproduce the build of the software. This includes the pristine source, any modifications required to build, shell commands to build/install it, and the meta-data. So, pick up the .src.rpm file from the URL you mention above. It's got everything you need... You can either install it and find the .spec file in /usr/src/redhat/SPECS, or you can extract the .src.rpm using "rpm2cpio" and using CPIO to get just the files you want. >I've asked Sean to contribute his specfile, but he hasn't given them >to me yet. Well, at some point we were talking about wether to eliminate the patches and how to do it for expat and ... I was actually thinking last night of putting them into CVS (I made some modifications that I thought would let Zope 2.4.3 build, but didn't and I backed them out). How would you like to deal with the .spec file and patches? I can easily enough turn the patches into sed commands in the setup, which would mean you could build the RPMs from the Python tar file directly, if included there. Do you want me to just mail the new .spec file to you when I ask you to upload the new RPMs, or do you want a script that would check out the latest .spec file from my CVS into your tree, update the version number in it, and go from there, or can I get access to your CVS for checking in new .specs? Or, I guess we could get it updated into the current CVS and I could submit patches through the Sourceforge. Except I always sourceFORGET to click the "click here to attach file" button. <sigh> Comments? Sean -- "McGuyver stole all his tricks from Dr. Who." Sean Reifschneider, Inimitably Superfluous <jafo(a)tummy.com> tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python

3 3

Re: [Python-Dev] Unicode strings as filenames
by Jack Jansen Jan. 5, 2002

Jan. 5, 2002

Recently, "M.-A. Lemburg" <mal(a)lemburg.com> said: > Jack Jansen wrote: > > > > Off on a slight tangent: > > On Mac OS X the default 8-bit encoding is UTF8. os.listdir() handles > > this fine and so does open(). The OS does all the hard work for > > you [...] > > But in Python (unix-Python we're talking here, not MacPython), > > unicode(filename) fails, because site.encoding is "ascii". > > > > Would it be safe to set site.encoding to utf8 on Mac OS X by default? > > I'd rather suggest to use UTF-8 as default encoding in the > subsystem layer I was talking about. Uhm... Do you mean Py_FileSystemDefaultEncoding? Otherwise: what do you mean? And, if you do mean Py_FSDE, would that also work for listdir()? No, I guess it can't because listdir() returns simple strings, so by the time I pass them to unicode() all knowledge that they came from listdir is gone... Hmm, shouldn't StringObjects themselves carry an encoding field (defaulting to sys.encoding)? That would solve quite a few issues. read() from a binary file would return the special encoding "binary", for instance, and then the "u" and "u#" formats could make a distinction between character strings (which would be converted to unicode using the encoding they carry) and binary strings (which would be interpreted as 16-bit chars). But interning may be a showstopper, now that I think of it... > Making UTF-8 the default Python system encoding would have many other > consequences -- and you'd probably lose a great deal of portability > since UTF-8 conversion (nearly) always will succeed while ASCII can > easily fail on other systems which use e.g. Latin-1 as native > encoding. What are your reasons for asserting this? If I read this correctly this would make Python compatible to the least common denominator of all platforms, while I think I would prefer it to allow access to all the niceties a platform gives. On Unix you really don't have a good guess for the encoding, but on MacOS and Windows you do... -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

2 1

RPM *.spec file
by Marek Pętlicki Jan. 5, 2002

Jan. 5, 2002

Hi! Could someone point me to the RPM *.spec file with which was built the Python 2.2 (ftp://ftp.python.org/pub/python/2.2/rpms/)? I've already installed from source, but I would like make some order in my system, and currently I have 3 versions of Python in various dirs :-) Q: why is the specfile not distributed with the sources? I've found some outdated BeOpen specfile, but it doesn't work. Yes, I can fix it, but what for when somewhere there seems to exist a fixed one, since *.rpm binaries are supported on ftp.python.org? bye -- Marek Pętlicki <marpet(a)linuxpl.org> Linux User ID=162988

3 5

Re: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Objec ts fileobject.c,2.141,2.142
by Anthony Baxter Jan. 4, 2002

Jan. 4, 2002

>>> "Martin v. Loewis" wrote > Yes. But it also tightens the behaviour, so it should not be applied > to maintainance branches: no correct program would work better with > this patch, but currently broken programs may stop working. Yep, what he said. It's a bug, but it doesn't cause programs that are broken to work. It does potentially change how they break, though - this has been one of the criterion I've been using to say "nope". -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

1 0

Re [Python-Dev] object equality vs identity, in and dicts idioms and speed
by Samuele Pedroni Jan. 4, 2002

Jan. 4, 2002

[Tim Peters] > Mapping what to what? A fine implementation of id() would be to hand each > new object a unique Java int from a global counter, incremented once per > Python object creation -- or a Java long if any JVM stays up long enough > that 32 bits is an issue <wink>. The problem are java class instances, sir, we use non-unique wrappers for them and identity is simulated. We could use a table to make the wrappers unique but we have potentially lots of them as you can imagine, jython people actually use java classes <wink>. So the workaround is to keep a table just for the java instances for which someone has asked the id. We cannot win-win so we try not to lose-lose. For simplicity we will extend the table approach to everything. If you have a win-win solution also in this case please ... Our goal is compatibility but we will suggest to avoid id as far as possible for production jython code ... regards

4 5

weave -- inline C/C++ in Python, an implementation
by eric Jan. 4, 2002

Jan. 4, 2002

Hello group, I'm pretty close to releasing weave 0.2, a tool that helps in combining C/C++ with Python code. There are basically three ways to use it. inline() offers inline C/C++ in Python. blitz() converts Python Numeric expression to C++ for fast execution. And, ext_tools offer a couple of classes that build extension classes. If you have a few cycles to spare, I'd appreciate a few eye balls on the documentation page and source. Also, if people could download the zip/exe/tar.gz files and let me know of any failures that would be helpful. The website provides info on how to test (it's very simple). Also, success reports on platforms would be good . W2K, RH 7.1, Debian are about all that has been tested. Here is the link: http://www.scipy.org/site_content/weave thanks, eric

1 0

Re: [Python-Dev] Unicode strings as filenames
by Jack Jansen Jan. 4, 2002

Jan. 4, 2002

Off on a slight tangent: On Mac OS X the default 8-bit encoding is UTF8. os.listdir() handles this fine and so does open(). The OS does all the hard work for you: it knows that some mounted disks may be in other 8-bit encodings (such as MacRoman or MacJapanese for old mac disks, or probably latin-1 for NFS filesystems, or god-knows-what for SMB mounted disks) and handles the conversion. But in Python (unix-Python we're talking here, not MacPython), unicode(filename) fails, because site.encoding is "ascii". Would it be safe to set site.encoding to utf8 on Mac OS X by default? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

3 2