Mailman 3 September 2008 - Python-Dev

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by "Martin v. Löwis" Oct. 1, 2008

Oct. 1, 2008

>> On Windows, we might reject bytes filenames for all file operations: open(), >> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) > > Since I've seen no objections to this yet: please no. If we offer a > "lower-level" bytes filename API, it should work for all platforms. Unfortunately, it can't. You cannot represent all possible file names in a byte string in Windows (just as you can't do so in a Unicode string on Unix). So using byte strings on Windows would work for some files, but fail for others. In particular, listdir might give you a list of file names which you then can't open/stat/recurse into. (of course, you could use UTF-8 as the file system encoding on Windows, but then you will have to rewrite a lot of C code first) Regards, Martin

6 8

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Oct. 1, 2008

Oct. 1, 2008

On Tue, Sep 30, 2008 at 12:42 PM, Terry Reedy <tjreedy(a)udel.edu> wrote: > Guido van Rossum wrote: >> >> On Tue, Sep 30, 2008 at 11:13 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: >>> >>> Victor Stinner schrieb: >>>> >>>> On Windows, we might reject bytes filenames for all file operations: >>>> open(), >>>> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) >>> >>> Since I've seen no objections to this yet: please no. If we offer a >>> "lower-level" bytes filename API, it should work for all platforms. >> >> I'm not sure either way. I've heard it claim that Windows filesystem >> APIs use Unicode natively. Does Python 3.0 on Windows currently >> support filenames expressed as bytes? Are they encoded first before >> passing to the Unicode APIs? Using what encoding? > > In 3.0rc1, the listdir doc needs updating: > "os.listdir(path) > Return a list containing the names of the entries in the directory. The list > is in arbitrary order. It does not include the special entries '.' and '..' > even if they are present in the directory. Availability: Unix, Windows. > > On Windows NT/2k/XP and Unix, if path is a Unicode object, the result will > be a list of Unicode objects." > > s/Unicode/bytes/ at least for Windows. > >>>> os.listdir(b'.') > [b'countries.txt', b'multeetest.py', b't1.py', b't1.pyc', b't2.py', b'tem', > b'temp.py', b'temp.pyc', b'temp2.py', b'temp3.py', b'temp4.py', b'test.py', > b'z', b'z.txt'] > > The bytes names do not work however: > >>>> t=open(b'tem') > Traceback (most recent call last): > File "<pyshell#23>", line 1, in <module> > t=open(b'tem') > File "C:\Programs\Python30\lib\io.py", line 284, in __new__ > return open(*args, **kwargs) > File "C:\Programs\Python30\lib\io.py", line 184, in open > raise TypeError("invalid file: %r" % file) > TypeError: invalid file: b'tem' > > Is this what you were asking? No, that's because bytes is missing from the explicit list of allowable types in io.open. Victor has a one-line trivial patch for this. Could you try this though? >>> import _fileio >>> _fileio._FileIO(b'tem') -- --Guido van Rossum (home page: http://www.python.org/~guido/)

2 1

Filename as byte string in python 2.6 or 3.0?
by Victor Stinner Oct. 1, 2008

Oct. 1, 2008

Hi, I read that Python 2.6 is planned to Wednesday. One bug is is still open and important for me: Python 2.6/3.0 are unable to use filename as byte strings. http://bugs.python.org/issue3187 The problem =========== On Windows, all filenames are unicode strings (I guess UTF-16-LE), but on UNIX for historical reasons, filenames are byte strings. On Linux, you can expect UTF-8 valid filenames but sometimes (eg. copy from a FAT32 USB key to an ext3 filesystem) you get invalid filename (byte string in a different charset than your default filesystem encoding (utf8)). Python functions using filenames ================================ In Python, you have (incomplete list): - filename producer: os.listdir(), os.walk(), glob.glob() - filename manipulation: os.path.*() - access file: open(), os.unlink(), shutil.rmtree() If you give unicode to producer, they return unicode _or_ byte strings (type may change for each filename :-/). Guido proposed to break this behaviour: raise an exception if unicode conversion fails. We may consider an option like "skip invalid". If you give bytes to producer, they only return byte strings. Great. Filename manipulation: in python 2.6/3.0, os.path.*() is not compatible with the type "bytes". So you can not use os.path.join(<your unicode path>, <bytes filename>) *nor* os.path.join(<your bytes path>, <bytes filename>) because os.path.join() (eg. with the posix version) uses path.endswith('/'). Access file: open() rejects the type bytes (it's just a test, open() supports bytes if you remove the test). As I remember, unlink() is compatible with bytes. But rmtree() fails because it uses os.path.join() (even if you give bytes directory, join() fails). Solutions ========= - producer: unicode => *only* unicode // bytes => bytes - manipulation: support both unicode and bytes but avoid (when it's possible) to mix bytes and characters - open(): allow bytes I implemented these solutions as a patch set attached to the issue #3187: * posix_path_bytes.patch: fix posixpath.join() to support bytes * io_byte_filename.patch: open() allows bytes filename * fnmatch_bytes.patch: patch fnmatch.filter() to accept bytes filenames * glob1_bytes.patch: fix glob.glob() to accept invalid directory name Mmmh, there is no patch for stop os.listdir() on invalid filename. Priority ======== I think that the problem is important because it's a regression from 2.5 to 2.6/3.0. Python 2.5 uses bytes filename, so it was possible to open/unlink "invalid" unicode strings (since it's not unicode but bytes). Well, if it's too late for the final versions, this problem should be at least fixed quickly. Test the problem ================ Example to create invalid filenames on Linux: $ mkdir /tmp/test $ cd /tmp/test $ touch $(echo -e "a\xffb") $ mkdir $(echo -e "dir\xffname") $ touch $(echo -e "dir\xffname/file") $ find . ./a?b ./dir?name ./dir?name/file Python 2.5: >>> import os >>> os.listdir('.') ['a\xffb', 'dir\xffname'] >>> open(os.listdir('.')[0]).close() # open file: ok >>> os.unlink(os.listdir('.')[0]) # remove file: ok >>> os.listdir('.') ['dir\xffname'] >>> shutil.rmtree(os.listdir('.')[0]) # remove dir: ok Wrong solutions =============== New type -------- I proposed an ugly type "InvalidFilename" mixing bytes and characters. As everybody using unicode knows, it's a bad idea :-) (and it introduces a new type). Convert bytes to unicode (replace) ---------------------------------- unicode_filename = unicode(bytes_filename, charset, "replace") Ok, you will get valid unicode strings which can be used in os.path.join() & friends, but open() or unlink() will fails because this filename doesn't exist! -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

17 38

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by "Martin v. Löwis" Sept. 30, 2008

Sept. 30, 2008

> Oh, ok. I had assumed Windows just uses a fixed encoding without the problem > of misencoded filenames. It's the other way 'round: On Windows, Unicode file names are the natural choice, and byte strings have limitations. In a sense, Windows got it right - but then, they started later. Unix missed the opportunity of declaring that all file APIs are UTF-8 (except for Plan-9 and OS X, neither being "true" Unix). Regards, Martin

2 3

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Sept. 30, 2008

Sept. 30, 2008

On Mon, Sep 29, 2008 at 11:22 PM, Georg Brandl <g.brandl(a)gmx.net> wrote: > No, that was not what I meant (although it is another possibility). As I wrote, > Martin's proposal that I support here is using the modified UTF-8 codec that > successfully roundtrips otherwise invalid UTF-8 data. I thought that the "successful rountripping" pretty much stopped as soon as the unicode data is exported to somewhere else -- doesn't it contain invalid surrogate sequences? In general, I'm very reluctant to use utf-8b given that it doesn't seem to be well documented as a standard anywhere. Providing some minimal APIs that can process raw-bytes filenames still makes more sense -- it is mostly analogous of our treatment of text files, where the underlying binary data is also accessible. > You seem to forget that (disregarding OSX here, since it already enforces > UTF-8) the majority of file names on Posix systems will be encoded correctly. Apparently under certain circumstances (external FS mounted) OSX can also have non-UTF-8 filenames. [...] > With the filenames decoded by UTF-8, your files named têste, ô, dossié will > be displayed and handled correctly. The others are *invalid* in the filesystem > encoding UTF-8 and therefore would be represented by something like > > u'dir\uXXffname' where XX is some private use Unicode namespace. It won't look > pretty when printed, but then, what do other applications do? They e.g. display > a question mark as you show above, which is not better in terms of readability. > > But it will work when given to a filename-handling function. Valid filenames > can be compared to Unicode strings. > > A real-world example: OpenOffice can't open files with invalid bytes in their > name. They are displayed in the "Open file" dialog, but trying to open fails. > This regularly drives me crazy. Let's not make Python not work this way too, > or, even worse, not even display those filenames. How can it *regularly* drive you crazy when "the majority of fie names [...] encoded correctly" (as you assert above)? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

4 8

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Marcin 'Qrczak' Kowalczyk Sept. 30, 2008

Sept. 30, 2008

2008/9/30 Glenn Linderman <v+python(a)g.nevcal.com>: > So the problem is that a Unicode file system interface can't deal with > non-UTF-8 byte streams as file names. > > So it seems there are four suggested approaches, all of which have aspects > that are inconvenient. Let's not forget what happens when a non-UTF-8 file name is read from a file or written to a file, under the assumption that the filename is written to the file directly (which probably breaks for filenames containing newlines or such). > 4) Use of bytes APIs on FS interfaces. This seems to be the "solution" > adopted by Posix that creates the "problem" encountered by Unicode-native > applications. It is cumbersome to deal with within applications that > attempt to display the names. What do Posix-style "open file" dialog boxes > do in this case? http://library.gnome.org/devel/glib/stable/glib-Character-Set-Conversion.ht… I used to observe three different ways to display such filenames within gedit (including %xx and \xx escapes), but now it is consistent, probably because it switched to using the above function everywhere: $ touch $'abc\xffz' $ gedit The Open dialog shows: abc�z (invalid encoding) When the file is open, the window title and the tab title show: abc�z and the same is in recent file list. It has a bug: it appends " (invalid encoding)" even if the filename contains a correctly encoded U+FFFD character. Nautilus has the same behavior and the same bug because this is a design bug of that function which does not allow to tell whether the conversion was successful. A filename containing a newline is sometimes displayed in two lines, and sometimes with a U+000A character from a fallback font (hex character number in a box). -- Marcin Kowalczyk qrczak(a)knm.org.pl http://qrnik.knm.org.pl/~qrczak/

1 0

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Sept. 30, 2008

Sept. 30, 2008

On Tue, Sep 30, 2008 at 1:12 PM, Terry Reedy <tjreedy(a)udel.edu> wrote: > Terry Reedy wrote: >> >> Guido van Rossum wrote: > >>> I'm not sure either way. I've heard it claim that Windows filesystem >>> APIs use Unicode natively. Does Python 3.0 on Windows currently >>> support filenames expressed as bytes? Are they encoded first before >>> passing to the Unicode APIs? Using what encoding? > >> [os.listdir(bytes) returns list of bytes, open(bytes) fails] > > More: > > The path functions seem also do not work: > >>>> op.abspath(b'tem') > ... > path = path.replace("/", "\\") > TypeError: expected an object with the buffer interface > > The error message is a bit cryptic given that the problem is that the > arguments to replace should be bytes instead of strings for a bytes path. > > .basename fails with > ... > while i and p[i-1] not in '/\\': > TypeError: 'in <string>' requires string as left operand, not int > > os.rename, os.stat, os.mkdir, os.rmdir work. I presume same is true for > others that normally work on windows. It looks roughly like the system calls do support bytes (using what encoding?) but the Python code in os.path doesn't. This is the same as the status quo on Linux. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Sept. 30, 2008

Sept. 30, 2008

On Tue, Sep 30, 2008 at 11:13 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: > Victor Stinner schrieb: >> On Windows, we might reject bytes filenames for all file operations: open(), >> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) > > Since I've seen no objections to this yet: please no. If we offer a > "lower-level" bytes filename API, it should work for all platforms. I'm not sure either way. I've heard it claim that Windows filesystem APIs use Unicode natively. Does Python 3.0 on Windows currently support filenames expressed as bytes? Are they encoded first before passing to the Unicode APIs? Using what encoding? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

2 1

Re: [Python-Dev] Patch for an initial support of bytes filename in Python3
by Bill Janssen Sept. 30, 2008

Sept. 30, 2008

Victor Stinner <victor.stinner(a)haypocalc.com> wrote: > - listdir(unicode) -> only unicode, *skip* invalid filenames > (as asked by Guido) Is there an option listdir(bytes) which will return *all* filenames (as byte sequences)? Otherwise, this seems troubling to me; *something* should be returned for filenames which can't be represented, even if it's only None. Bill

2 3

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Sept. 30, 2008

Sept. 30, 2008

> Victor Stinner schrieb: (Thanks Victor for moving this to the list. Having a discussion in the tracker is really painful, I find.) >> POSIX OS >> -------- >> >> The default behaviour should be to use unicode and raise an error if >> conversion to unicode fails. It should also be possible to use bytes using >> bytes arguments and optional arguments (for getcwd). >> >> - listdir(unicode) -> unicode and raise an error on invalid filename I know I keep flipflopping on this one, but the more I think about it the more I believe it is better to drop those names than to raise an exception. Otherwise a "naive" program that happens to use os.listdir() can be rendered completely useless by a single non-UTF-8 filename. Consider the use of os.listdir() by the glob module. If I am globbing for *.py, why should the presence of a file named b'\xff' cause it to fail? Robust programs using os.listdir() should use the bytes->bytes version. >> - listdir(bytes) -> bytes >> - getcwd() -> unicode >> - getcwd(bytes=True) -> bytes >> - open(): accept bytes or unicode >> >> os.path.*() should accept operations on bytes filenames, but maybe not on >> bytes+unicode arguments. os.path.join('directory', b'filename'): raise an >> error (or use *implicit* conversion to bytes)? (Yeah, it should be all bytes or all strings.) On Mon, Sep 29, 2008 at 9:45 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: > This approach (changing all path-handling functions to accept either bytes > or string, but not both) is doomed in my eyes. First, there are lots of them, > second, they are not only in os.path but in many modules and also in user > code, and third, I see no clean way of implementing them in the specified way. > (Just try to do it with os.path.join as an example; I couldn't find the > good way to write it, only the bad and the ugly...) It doesn't have to be supported for all operations -- just enough to be able to access all the system calls. and do the most basic pathname manipulations (split and join -- almost everything else can be built out of those). > If I had to choose, I'd still argue for the modified UTF-8 as filesystem > encoding (if it were UTF-8 otherwise), despite possible surprises when a > such-encoded filename escapes from Python. I'm having a hard time finding info about UTF-8b. Does anyone have a decent link? I noticed that OSX has a different approach yet. I believe it insists on valid UTF-8 filenames. It may even require some normalization but I don't know if the kernel enforces this. I tried to create a file named b'\xff' and it came out as %ff. Then "rm %ff" worked. So I think it may be replacing all bad UTF8 sequences with their % encoding. The "set filesystem encoding to be Latin-1" approach has a certain charm as well, but clearly would be a mistake on OSX, and probably on other systems too (whenever the user doesn't think in Latin-1). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

9 12