Mailman 3 September 2008 - Python-Dev

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by "Martin v. Löwis" Oct. 1, 2008

Oct. 1, 2008

>> On Windows, we might reject bytes filenames for all file operations: open(), >> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) > > Since I've seen no objections to this yet: please no. If we offer a > "lower-level" bytes filename API, it should work for all platforms. Unfortunately, it can't. You cannot represent all possible file names in a byte string in Windows (just as you can't do so in a Unicode string on Unix). So using byte strings on … [View More]

6 8

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Oct. 1, 2008

Oct. 1, 2008

On Tue, Sep 30, 2008 at 12:42 PM, Terry Reedy <tjreedy(a)udel.edu> wrote: > Guido van Rossum wrote: >> >> On Tue, Sep 30, 2008 at 11:13 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: >>> >>> Victor Stinner schrieb: >>>> >>>> On Windows, we might reject bytes filenames for all file operations: >>>> open(), >>>> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) >>> >>> … [View More]

2 1

Filename as byte string in python 2.6 or 3.0?
by Victor Stinner Oct. 1, 2008

Oct. 1, 2008

Hi, I read that Python 2.6 is planned to Wednesday. One bug is is still open and important for me: Python 2.6/3.0 are unable to use filename as byte strings. http://bugs.python.org/issue3187 The problem =========== On Windows, all filenames are unicode strings (I guess UTF-16-LE), but on UNIX for historical reasons, filenames are byte strings. On Linux, you can expect UTF-8 valid filenames but sometimes (eg. copy from a FAT32 USB key to an ext3 filesystem) you get invalid filename (… [View More]byte string in a different charset than your default filesystem encoding (utf8)). Python functions using filenames ================================ In Python, you have (incomplete list): - filename producer: os.listdir(), os.walk(), glob.glob() - filename manipulation: os.path.*() - access file: open(), os.unlink(), shutil.rmtree() If you give unicode to producer, they return unicode _or_ byte strings (type may change for each filename :-/). Guido proposed to break this behaviour: raise an exception if unicode conversion fails. We may consider an option like "skip invalid". If you give bytes to producer, they only return byte strings. Great. Filename manipulation: in python 2.6/3.0, os.path.*() is not compatible with the type "bytes". So you can not use os.path.join(<your unicode path>, <bytes filename>) *nor* os.path.join(<your bytes path>, <bytes filename>) because os.path.join() (eg. with the posix version) uses path.endswith('/'). Access file: open() rejects the type bytes (it's just a test, open() supports bytes if you remove the test). As I remember, unlink() is compatible with bytes. But rmtree() fails because it uses os.path.join() (even if you give bytes directory, join() fails). Solutions ========= - producer: unicode => *only* unicode // bytes => bytes - manipulation: support both unicode and bytes but avoid (when it's possible) to mix bytes and characters - open(): allow bytes I implemented these solutions as a patch set attached to the issue #3187: * posix_path_bytes.patch: fix posixpath.join() to support bytes * io_byte_filename.patch: open() allows bytes filename * fnmatch_bytes.patch: patch fnmatch.filter() to accept bytes filenames * glob1_bytes.patch: fix glob.glob() to accept invalid directory name Mmmh, there is no patch for stop os.listdir() on invalid filename. Priority ======== I think that the problem is important because it's a regression from 2.5 to 2.6/3.0. Python 2.5 uses bytes filename, so it was possible to open/unlink "invalid" unicode strings (since it's not unicode but bytes). Well, if it's too late for the final versions, this problem should be at least fixed quickly. Test the problem ================ Example to create invalid filenames on Linux: $ mkdir /tmp/test $ cd /tmp/test $ touch $(echo -e "a\xffb") $ mkdir $(echo -e "dir\xffname") $ touch $(echo -e "dir\xffname/file") $ find . ./a?b ./dir?name ./dir?name/file Python 2.5: >>> import os >>> os.listdir('.') ['a\xffb', 'dir\xffname'] >>> open(os.listdir('.')[0]).close() # open file: ok >>> os.unlink(os.listdir('.')[0]) # remove file: ok >>> os.listdir('.') ['dir\xffname'] >>> shutil.rmtree(os.listdir('.')[0]) # remove dir: ok Wrong solutions =============== New type -------- I proposed an ugly type "InvalidFilename" mixing bytes and characters. As everybody using unicode knows, it's a bad idea :-) (and it introduces a new type). Convert bytes to unicode (replace) ---------------------------------- unicode_filename = unicode(bytes_filename, charset, "replace") Ok, you will get valid unicode strings which can be used in os.path.join() & friends, but open() or unlink() will fails because this filename doesn't exist! -- Victor Stinner aka haypo http://www.haypocalc.com/blog/ [View Less]

17 38

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by "Martin v. Löwis" Sept. 30, 2008

Sept. 30, 2008

> Oh, ok. I had assumed Windows just uses a fixed encoding without the problem > of misencoded filenames. It's the other way 'round: On Windows, Unicode file names are the natural choice, and byte strings have limitations. In a sense, Windows got it right - but then, they started later. Unix missed the opportunity of declaring that all file APIs are UTF-8 (except for Plan-9 and OS X, neither being "true" Unix). Regards, Martin

2 3

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Sept. 30, 2008

Sept. 30, 2008

On Mon, Sep 29, 2008 at 11:22 PM, Georg Brandl <g.brandl(a)gmx.net> wrote: > No, that was not what I meant (although it is another possibility). As I wrote, > Martin's proposal that I support here is using the modified UTF-8 codec that > successfully roundtrips otherwise invalid UTF-8 data. I thought that the "successful rountripping" pretty much stopped as soon as the unicode data is exported to somewhere else -- doesn't it contain invalid surrogate sequences? In general, I'm … [View More]

4 8

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Marcin 'Qrczak' Kowalczyk Sept. 30, 2008

Sept. 30, 2008

2008/9/30 Glenn Linderman <v+python(a)g.nevcal.com>: > So the problem is that a Unicode file system interface can't deal with > non-UTF-8 byte streams as file names. > > So it seems there are four suggested approaches, all of which have aspects > that are inconvenient. Let's not forget what happens when a non-UTF-8 file name is read from a file or written to a file, under the assumption that the filename is written to the file directly (which probably breaks for filenames … [View More]

1 0

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Sept. 30, 2008

Sept. 30, 2008

On Tue, Sep 30, 2008 at 1:12 PM, Terry Reedy <tjreedy(a)udel.edu> wrote: > Terry Reedy wrote: >> >> Guido van Rossum wrote: > >>> I'm not sure either way. I've heard it claim that Windows filesystem >>> APIs use Unicode natively. Does Python 3.0 on Windows currently >>> support filenames expressed as bytes? Are they encoded first before >>> passing to the Unicode APIs? Using what encoding? > >> [os.listdir(bytes) returns list … [View More]

1 0

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Sept. 30, 2008

Sept. 30, 2008

On Tue, Sep 30, 2008 at 11:13 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: > Victor Stinner schrieb: >> On Windows, we might reject bytes filenames for all file operations: open(), >> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) > > Since I've seen no objections to this yet: please no. If we offer a > "lower-level" bytes filename API, it should work for all platforms. I'm not sure either way. I've heard it claim that Windows filesystem APIs … [View More]

2 1

Re: [Python-Dev] Patch for an initial support of bytes filename in Python3
by Bill Janssen Sept. 30, 2008

Sept. 30, 2008

Victor Stinner <victor.stinner(a)haypocalc.com> wrote: > - listdir(unicode) -> only unicode, *skip* invalid filenames > (as asked by Guido) Is there an option listdir(bytes) which will return *all* filenames (as byte sequences)? Otherwise, this seems troubling to me; *something* should be returned for filenames which can't be represented, even if it's only None. Bill

2 3

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Sept. 30, 2008

Sept. 30, 2008

> Victor Stinner schrieb: (Thanks Victor for moving this to the list. Having a discussion in the tracker is really painful, I find.) >> POSIX OS >> -------- >> >> The default behaviour should be to use unicode and raise an error if >> conversion to unicode fails. It should also be possible to use bytes using >> bytes arguments and optional arguments (for getcwd). >> >> - listdir(unicode) -> unicode and raise an error on invalid filename … [View More]I know I keep flipflopping on this one, but the more I think about it the more I believe it is better to drop those names than to raise an exception. Otherwise a "naive" program that happens to use os.listdir() can be rendered completely useless by a single non-UTF-8 filename. Consider the use of os.listdir() by the glob module. If I am globbing for *.py, why should the presence of a file named b'\xff' cause it to fail? Robust programs using os.listdir() should use the bytes->bytes version. >> - listdir(bytes) -> bytes >> - getcwd() -> unicode >> - getcwd(bytes=True) -> bytes >> - open(): accept bytes or unicode >> >> os.path.*() should accept operations on bytes filenames, but maybe not on >> bytes+unicode arguments. os.path.join('directory', b'filename'): raise an >> error (or use *implicit* conversion to bytes)? (Yeah, it should be all bytes or all strings.) On Mon, Sep 29, 2008 at 9:45 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: > This approach (changing all path-handling functions to accept either bytes > or string, but not both) is doomed in my eyes. First, there are lots of them, > second, they are not only in os.path but in many modules and also in user > code, and third, I see no clean way of implementing them in the specified way. > (Just try to do it with os.path.join as an example; I couldn't find the > good way to write it, only the bad and the ugly...) It doesn't have to be supported for all operations -- just enough to be able to access all the system calls. and do the most basic pathname manipulations (split and join -- almost everything else can be built out of those). > If I had to choose, I'd still argue for the modified UTF-8 as filesystem > encoding (if it were UTF-8 otherwise), despite possible surprises when a > such-encoded filename escapes from Python. I'm having a hard time finding info about UTF-8b. Does anyone have a decent link? I noticed that OSX has a different approach yet. I believe it insists on valid UTF-8 filenames. It may even require some normalization but I don't know if the kernel enforces this. I tried to create a file named b'\xff' and it came out as %ff. Then "rm %ff" worked. So I think it may be replacing all bad UTF8 sequences with their % encoding. The "set filesystem encoding to be Latin-1" approach has a certain charm as well, but clearly would be a mistake on OSX, and probably on other systems too (whenever the user doesn't think in Latin-1). -- --Guido van Rossum (home page: http://www.python.org/~guido/) [View Less]

9 12