Mailman 3 April 2009 - Python-Dev

PEP 383 (again)
by Thomas Breuel April 30, 2009

April 30, 2009

I thought PEP-383 was a fairly neat approach, but after thinking about it, I now think that it is wrong. PEP-383 attempts to represent non-UTF-8 byte sequences in Unicode strings in a reversible way. But how do those non-UTF-8 byte sequences get into those path names in the first place? Most likely because an encoding other than UTF-8 was used to write the file system, but you're now trying to interpret its path names as UTF-8. Quietly escaping a bad UTF-8 encoding with private Unicode characters is unlikely to be the right thing, since using the wrong encoding likely means that other characters are decoded incorrectly as well. As a result, the path name may fail in string comparisons and pattern matching, and will look wrong to the user in print statements and dialog boxes. Therefore, when Python encounters path names on a file system that are not consistent with the (assumed) encoding for that file system, Python should raise an error. If you really don't care what the string looks like and you just want an encoding that round-trips without loss, you can probably just set your encoding to one of the 8 bit encodings, like ISO 8859-15. Decoding arbitrary byte sequences to unicode strings as ISO 8859-15 is no less correct than decoding them as the proposed "utf-8b". In fact, the most likely source of non-UTF-8 sequences is ISO 8859 encodings. As for what the byte-oriented interfaces should do, they are simply platform dependent. On UNIX, they should do the obvious thing. On Windows, they can either hook up to the low-level byte-oriented system calls that the systems supply, or Windows could fake it and have the byte-oriented interfaces use UTF-8 encodings always and reject non-UTF-8 sequences as illegal (there are already many illegal byte sequences anyway). Tom

10 30

string to float containing whitespace
by skip＠pobox.com April 29, 2009

April 29, 2009

Someone please tell me I'm not going mad. I could have sworn that once upon a time attempting to convert numeric strings to ints or floats if they contained whitespace raised an exception. As far back as 1.5.2 it appears that float(), string.atof() and string.atoi() allow whitespace. Maybe I'm thinking of trailing non-numeric, non-whitespace characters. Skip

3 3

Installing Python 2.5.4 from Source under Windows
by Paul Franz April 29, 2009

April 29, 2009

I have looked and looked and looked. But I can not find any directions on how to install the version of Python build using Microsoft's compiler. It builds. I get the dlls and the exe's. But there is no documentation that says how to install what has been built. I have read every readme and stop by the IRC channel and there seems to be nothing. Any ideas where I can look? Paul Franz

2 2

Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
by Hrvoje Niksic April 29, 2009

April 29, 2009

Zooko O'Whielacronx wrote: >> If you switch to iso8859-15 only in the presence of undecodable >> UTF-8, then you have the same round-trip problem as the PEP: both >> b'\xff' and b'\xc3\xbf' will be converted to u'\u00ff' without a >> way to unambiguously recover the original file name. > > Why do you say that? It seems to work as I expected here: > > >>> '\xff'.decode('iso-8859-15') > u'\xff' > >>> '\xc3\xbf'.decode('iso-8859-15') > u'\xc3\xbf' Here is what I mean by "switch to iso8859-15" only in the presence of undecodable UTF-8: def file_name_to_unicode(fn, encoding): try: return fn.decode(encoding) except UnicodeDecodeError: return fn.decode('iso-8859-15') Now, assume a UTF-8 locale and try to use it on the provided example file names. >>> file_name_to_unicode(b'\xff', 'utf-8') 'ÿ' >>> file_name_to_unicode(b'\xc3\xbf', 'utf-8') 'ÿ' That is the ambiguity I was referring to -- to different byte sequences result in the same unicode string.

1 0

Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
by Hrvoje Niksic April 29, 2009

April 29, 2009

Thomas Breuel wrote: > But the biggest problem with the proposal is that it isn't needed: if > you want to be able to turn arbitrary byte sequences into unicode > strings and back, just set your encoding to iso8859-15. That already > works and it doesn't require any changes. Are you proposing to unconditionally encode file names as iso8859-15, or to do so only when undecodeable bytes are encountered? If you unconditionally set encoding to iso8859-15, then you are effectively reverting to treating file names as bytes, regardless of the locale. You're also angering a lot of European users who expect iso8859-2, etc. If you switch to iso8859-15 only in the presence of undecodable UTF-8, then you have the same round-trip problem as the PEP: both b'\xff' and b'\xc3\xbf' will be converted to u'\u00ff' without a way to unambiguously recover the original file name.

3 2

Re: [Python-Dev] PEP 383 (again)
by Hrvoje Niksic April 28, 2009

April 28, 2009

Lino Mastrodomenico wrote: > Let's suppose that I use Python 2.x or something else to create a file > with name b'\xff'. My (Linux) system has a sane configuration and the > filesystem encoding is UTF-8, so it's an invalid name but the kernel > will blindly accept it anyway. > > With this PEP, Python 3.1 listdir() will convert b'\xff' to the string '\udcff'. One question that really bothers me about this proposal is the following: Assume a UTF-8 locale. A file named b'\xff', being an invalid UTF-8 sequence, will be converted to the half-surrogate '\udcff'. However, a file named b'\xed\xb3\xbf', a valid[1] UTF-8 sequence, will also be converted to '\udcff'. Those are quite different POSIX pathnames; how will Python know which one it was when I later pass '\udcff' to open()? A poster hinted at this question, but I haven't seen it answered, yet. [1] I'm assuming that it's valid UTF8 because it passes through Python 2.5's '\xed\xb3\xbf'.decode('utf-8'). I don't claim to be a UTF-8 expert.

2 1

One more proposed formatting change for 3.1
by Mark Dickinson April 28, 2009

April 28, 2009

Here's one more proposed change, this time for formatting of floats using format() and the empty presentation type. To avoid repeating myself, here's the text from the issue I just opened: http://bugs.python.org/issue5864 """ In all versions of Python from 2.6 up, I get the following behaviour: >>> format(123.456, '.4') '123.5' >>> format(1234.56, '.4') '1235.0' >>> format(12345.6, '.4') '1.235e+04' The first and third results are as I expect, but the second is somewhat misleading: it gives 5 significant digits when only 4 were requested, and moreover the last digit is incorrect. I propose that Python 2.7 and Python 3.1 be changed so that the output for the second line above is '1.235e+03'. """ This issue seems fairly clear cut to me, and I doubt that there's been enough uptake of 'format' yet for this to risk significant breakage. So unless there are objections I'll plan to make this change before this weekend's beta. Mark

2 1

Can not run under python 2.6
by Jianchun Zhou April 28, 2009

April 28, 2009

Hi, there: I am new to python, and now I got a trouble: I have an application named canola, it is written under python 2.5, and can run normally under python 2.5 But when it comes under python 2.6, problem up, it says: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/terra/core/plugin_manager.py", line 151, in _load_plugins classes = plg.load() File "/usr/lib/python2.6/site-packages/terra/core/plugin_manager.py", line 94, in load mod = self._ldr.load() File "/usr/lib/python2.6/site-packages/terra/core/module_loader.py", line 42, in load mod = __import__(modpath, fromlist=[mod_name]) ImportError: Import by filename is not supported. Any body any idea what should I do? -- Best Regards

2 2

2.6.2 Vista installer failure on upgrade from 2.6.1
by Jim Kleckner April 27, 2009

April 27, 2009

I went to upgrade a Vista machine from 2.6.1 to 2.6.2 and got error 2755 with the message "system cannot open the device or file". I uninstalled 2.6.1, removing all residual files also, and got the error message again. When I ran msiexec as follows to get a log, it magically worked: msiexec /i python-2.6.2.msi /l*v install.log Should I attempt to explore this further or just be happy?

2 1

UTF-8 Decoder
by Jeroen Ruigrok van der Werven April 27, 2009

April 27, 2009

[Note: I haven't looked thoroughly at our handling yet, so hence I raise the question.] This got posted on the Unicode list, does it seem interesting for Python itself, the UTF-8 to UTF-16 transcoding might be? http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーンラウフロックヴァンデルウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Whenever you meet difficult situations dash forward bravely and joyfully...

2 3