I can't agree here. File handling is a fundamental operation and I would expect something like:

...

...

...

for fname in os.listdir('.'): ... if os.path.isfile(fname): ... file(fname)

to work for all files.

I agree. However, if it fails: is the bug of the Python, or of the system administrator maintaining it?

To have to know to put in special handling for certain corner case filenames or worse to not be able to open some files at all would be a serious loss. It would also complicate migrating code correctly to 3.0.

I agree completely. Unfortunately, all proposed solutions *do* require special handling for certain corner cases.

Regardless of whose fault the underlying issue is, someone has to deal with the problem and if core Python doesn't, each developer who encounters the problem will have to come up with his/her own solution.

This is quite in the abstract. Can you be more specific? Regards, Martin

On 9/27/08, "Martin v. Löwis" wrote:

...

I think that the problem is important because it's a regression from 2.5 to 2.6/3.0. Python 2.5 uses bytes filename, so it was possible to open/unlink "invalid" unicode strings (since it's not unicode but bytes).

I'd like to stress that the problem is *not* a regression from 2.5 to 2.6.

As for 3.0, I'd like to argue that the problem is a minor issue. Even though you may run into file names that can't be decoded, that happening really indicates some bigger problem in the management of the system where this happens, and the proper solution (IMO) should be to change the system (leaving open the question whether or not Python should be also changed to work with such broken systems).

Regards, Martin

Note: bcc python-dev,cc: python-3000 "broken" systems will always exist. Code to deal with them must be possible to write in python 3.0. since any given path (not just fs) can have its own encoding it makes the most sense to me to let the OS deal with the errors and not try to enforce bytes vs string encoding type at the python lib. level. -gps

On Sunday 28 September 2008, Gregory P. Smith wrote:

"broken" systems will always exist. Code to deal with them must be possible to write in python 3.0.

since any given path (not just fs) can have its own encoding it makes the most sense to me to let the OS deal with the errors and not try to enforce bytes vs string encoding type at the python lib. level.

Actually I'm afraid that that isn't really useful. I, too, would like to kick peoples' back in order to get the to fix their systems or use the proper codepage while mounting etc, etc, but that is not going to happen soon. Just ignoring those broken systems is tempting, but alienating a large group of users isn't IMHO worth it. Instead, I'd like to present a different approach: 1. For POSIX platforms (using a byte string for the path): Here, the first approach is to convert the path to Unicode, according to the locale's CTYPE category. Hopefully, it will be UTF-8, but also codepages should work. If there is a segment (a byte sequence between two path separators) where it doesn't work, it uses an ASCII mapping where possible and codepoints from the "Private Use Area" (PUA) of Unicode for the non-decodable bytes. In order to pass this path to fopen(), each segment would be converted to a byte string again, using the locale's CTYPE category except for segments which use the PUA where it simply encodes the original bytes. 2. For win32 platforms, the path is already Unicode (UTF-16) and the whole problem is solved or not solved by the OS. In the end, both approaches yield a path represented by a Unicode string for intermediate use, which provides maximum flexibility. Further, it preserves "broken" encodings by simply mapping their byte-values to the PUA of Unicode. Maybe not using a string to represent a path would be a good idea, too. At least it would make it very clear that the string is not completely free-form. Uli -- Sator Laser GmbH Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at http://www.satorlaser.de/ ************************************************************************************** Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht verantwortlich. **************************************************************************************

On Monday 29 September 2008, M.-A. Lemburg wrote:

On 2008-09-29 12:50, Ulrich Eckhardt wrote:

...

1. For POSIX platforms (using a byte string for the path): Here, the first approach is to convert the path to Unicode, according to the locale's CTYPE category. Hopefully, it will be UTF-8, but also codepages should work. If there is a segment (a byte sequence between two path separators) where it doesn't work, it uses an ASCII mapping where possible and codepoints from the "Private Use Area" (PUA) of Unicode for the non-decodable bytes. In order to pass this path to fopen(), each segment would be converted to a byte string again, using the locale's CTYPE category except for segments which use the PUA where it simply encodes the original bytes.

I'm not sure how this would work. How would you map the private use code points back to bytes ? Using a special codec that knows about these code points ? How would the fopen() know to use that special codec instead of e.g. the UTF-8 codec ?

Sorry, I wasn't clear enough. I'll try to explain further... Let's assume we have a filename like this: 0xc2 0xa9 0x2f 0x7f The first two bytes are the copyright sign encoded in UTF-8, followed by a slash (0x2f, path separator) and a character encoded in an unknown codepage (0x7f is not ASCII!). The first thing when receiving that path from the system would be to split it into segments, here we would get two of them, one with 0xc2 0xa9 and the other with 0x7f. This uses the fact that the separator (slash/0x2f) is rather universal (Note: I'm not sure about encodings like BIG5, i.e. ones that are neither UTF-8 nor derived from ASCII). For each segment, we would apply the locale's CTYPE facet and get the Unicode codepoint 0xa9 for the first segment, while the second one fails to convert. So, for the second one, we simply check for each byte if it is valid and printable ASCII (0x7f isn't). If it is, we emit the byte as Unicode codepoint. Otherwise, we map to the PUA. The PUA reserves 0xe000 to 0xf8ff for private uses. I would simply encode the byte 0x7f as 0xe07f, i.e. map it to the beginning of that range. Eventually, we would end up with the following Unicode codepoints: 0xa9, 0x2f, 0xe07f When converting to a byte string for use with fopen(), we simply inspect the supplied string again. If a segment contains elements of the PUA, we simply reverse the mapping for those and leave the others in that segment as-is. For all other segments, we apply the CTYPE conversion. Notes: * This effectively converts the current path representation (a string) into a sequence of segments where each segment can either be a fully Unicode-capable string or a raw byte string without any known interpretation. However, instead of using an array for that, it uses a string, which is what most people's code expects anyway. * You could also work on a byte-base instead of splitting the path in segments first. I just assumed that a single segment will not contain valid UTF-8 sequences mixed with invalid ones. A path however can contain both correctly and incorrectly encoded segments.

BTW: Private use areas in Unicode are meant for e.g. company specific code points. Using them for escaping purposes is likely to cause problems due to assignment clashes.

I'm not sure if the use I proposed is correct according to the intended use of the PUA. I know that ideally no such string would escape from Python, i.e. it should only be visible internally. I would guess that that is something the PUA was intended for. Uli -- Sator Laser GmbH Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at http://www.satorlaser.de/ ************************************************************************************** Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht verantwortlich. **************************************************************************************

Originally I thought that this was a valid idea, but then it became clear that this could be a problem. Consider a filename which includes a UTF-8 encoding of a PUA code point.

I still think it's a valid idea. For non-UTF-8 file system encodings, use PUA characters, and generate them through an error handler. If the file system encoding is UTF-8, use UTF-8b instead as the file system encoding.

Viewing the PUA with GNOME charmap, I can see that many code points there have character renderings on my Ubuntu system. I have to assume, therefore, that there are other (and potentially conflicting) uses for this unicode feature.

Depends on how you use it. If you use the PUA block 1 (i.e. U+E000..U+F8FF), there is a realistic chance of collision. If you use the Plane 15 or Plane 16 PUA blocks, there is currently zero chance of collision (AFAIK). PUA has a wide use for additional characters in TrueType, but I don't think many tools even support plane 15 and 16 for generating fonts, or rendering them (it may even that the TrueType/OpenType format doesn't support them in the first place). However, Python can make use of these planes fairly easily, even in 2-byte mode (through UTF-16). Regards, Martin

Le Tuesday 30 September 2008 01:31:45 Adam Olsen, vous avez écrit :

The alternative is not be valid unicode, but since we can't use such objects with external libs, can't even print them, we might as well call them something else. We already have a name for that: bytes.

:-)

On Tue, 2008-09-30 at 19:45 +1000, Nick Coghlan wrote:

To my mind, there are two kinds of app in the world when it comes to file paths: 1) "Normal" apps (e.g. a word processor), that are only interested in files with sane, well-formed file names that can be properly decoded to Unicode with the filesystem encoding identified by Python. If there is invalid data on the filesystem, they don't care and don't want to see it or have to deal with it.

I am not convinced that a word processor can just ignore files with (what it thinks are) undecodable file names. In countries with a history of incompatible national encodings, such file names crop up very often, sometimes as a natural consequence of data migrating from older systems to newer ones. You can and do encounter "invalid" file names in the filesystems of mainstream users even without them using buggy or obsolete software.

On Tue, 2008-09-30 at 07:26 -0700, Guido van Rossum wrote:

...

I am not convinced that a word processor can just ignore files with (what it thinks are) undecodable file names. In countries with a history of incompatible national encodings, such file names crop up very often, sometimes as a natural consequence of data migrating from older systems to newer ones. You can and do encounter "invalid" file names in the filesystems of mainstream users even without them using buggy or obsolete software.

This is a quality of implementation issue. Either the word processor is written to support "undecodable" files, or it isn't. If it isn't, there's nothing that can be done about it (short of buying another wordprocessor)

I agree with this. I just believe the underlying python APIs shouldn't make it impossible (or unnecessarily hard) for the word processor to implement showing of files with undecodable names. For example, implementing os.listdir to return the file names as Unicode subclasses with ability to access the underlying bytes (automatically recognized by open and friends) sounds like a good compromise that allows the word processor to both have the cake and eat it.

On Tue, Sep 30, 2008 at 11:12 AM, wrote:

The one thing it doesn't do is expose the decoding rules for the higher- level applications to deal with. I am pretty sure I don't understand how the interaction between filesystem encoding and user locale works in that case, though, so I can't immediately recommend a way to do it.

You can ask what the filesystem encoding is with sys.getfilesystemencoding(). On my Linux box I can make this return anything I like by setting LC_CTYPE=en_US.<whatever> (as long as <whatever> is a recognized encoding). There are probably 5 other environment variables to influence this. :-( Of course that doesn't help for undecodable filenames, and in that case I don't think *anything* can help you unless you have a lot of additional knowledge about what the user might be doing, e.g. you know a few other encodings to try that make sense for their environment. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Tue, Sep 30, 2008 at 11:42 AM, wrote:

There are other ways to glean this knowledge; for example, looking at the 'iocharset' or 'nls' mount options supplied to mount various filesystems. I thought maybe Python (or some C library call) might be invoking some logic that did something with data like that; if not, great, one day when I have some free time (meaning: never) I can implement that logic myself without duplicating a bunch of work.

I know we could do a better job, but absent anyone who knows what they're doing we've chosen a fairly conservative approach. I certainly hope that someone will contribute some mean encoding-guessing code to the stdlib that users can use. I'm not sure if I'll ever endorse doing this automatically in io.open(), though I'd be fine with a convention like passing encoding="guess". -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Tue, Sep 30, 2008 at 2:45 AM, Nick Coghlan wrote:

Adam Olsen wrote:

...

Lossy conversion just moves around what gets treated as garbage. As all valid unicode scalars can be round tripped, there's no way to create a valid unicode file name without being lossy. The alternative is not be valid unicode, but since we can't use such objects with external libs, can't even print them, we might as well call them something else. We already have a name for that: bytes.

To my mind, there are two kinds of app in the world when it comes to file paths: 1) "Normal" apps (e.g. a word processor), that are only interested in files with sane, well-formed file names that can be properly decoded to Unicode with the filesystem encoding identified by Python. If there is invalid data on the filesystem, they don't care and don't want to see it or have to deal with it. 2) "Filesystem" apps (e.g. a filesystem explorer), that need to be able to deal with malformed filenames that may not decode properly using the identified filesystem encoding.

For the former category of apps, the presence of a malformed filename should NOT disrupt the processing of well-formed files and directories. Those applications should "just work", even if the underlying filesystem has a few broken filenames.

Right. Totally agreed.

The latter category of applications need some way of defining their own application-specific handling of malformed names.

Agreed again.

That screams "callback" to me - and one mechanism to achieve that would be to expose the unicode "errors" argument for filesystem operations that return file paths (e.g. os.getcwd(), os.listdir(), os.readlink(), os.walk()).

Hm. This doesn't scream callback to me at all. I would never have thought of callbacks for this use case -- and I don't think it's a good idea. The callback would either be an extra argument to all system calls (bad, ugly etc., and why not go with the existing unicode encoding and error flags if we're adding extra args?) or would be global, where I'd be worried that it might interfere with the proper operation of library code that is several abstractions away from whoever installed the callback, not under their control, and not expecting the callback. I suppose I may have totally misunderstood your proposal, but in general I find callbacks unwieldy.

Once that was exposed, the existing error handling machinery in the codecs module could be used to allow applications to define their own custom error handling for Unicode decode errors in these operations. (e.g. set "codecs.register_error('bad_filepath', handle_filepath_error)", then use "errors='bad_filepath'" in the relevant os API calls)

The default handling could be left at "strict", with os.listdir() and os.walk() specifically ignoring path entries that trigger UnicodeDecodeError.

getcwd() and readlink() could just propagate the exception, since they have no other information to return.

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Tue, Sep 30, 2008 at 3:43 PM, Nick Coghlan wrote:

Guido van Rossum wrote:

...

The callback would either be an extra argument to all system calls (bad, ugly etc., and why not go with the existing unicode encoding and error flags if we're adding extra args?) or would be global, where I'd be worried that it might interfere with the proper operation of library code that is several abstractions away from whoever installed the callback, not under their control, and not expecting the callback.

I suppose I may have totally misunderstood your proposal, but in general I find callbacks unwieldy.

Not really - later in the email, I actually pointed out that exposing the unicode errors flag for the implicit PyUnicode_Decode invocations would be enough to enable a callback mechanism.

However, James's post pointing out that this is a problem that also affects environment variables and command line arguments, not just file paths completely kills any hope of purely callback based approach - that processing needs to "just work" without any additional intervention from the application.

Of the suggestions I've seen so far, I like Marcin's Mono-inspired NULL-escape codec idea the best. Since these strings all come from parts of the environment where NULLs are not permitted, a simple "'\0' in text" check will immediately identify any strings where decoding failed (for applications which care about the difference and want to try to do better), while applications which don't care will receive perfectly valid Python strings that can be passed around and manipulated as if the decoding error never happened.

It avoids the technical problems, but it's still magical behaviour that users have to learn, whereas bytes/unicode polymorphism uses the distinctions you should already know about. There's also a problem of how to turn it on. I'm against automatically Python changing the filesystem encoding, no matter how well intentioned. Better to let the app do that, which is easy and could be done for all apps (not just python!) if someone defined a libc encoding of "null-escaped UTF-8". On the whole I'm only -0 on it (compared to -1 for UTF-8b). -- Adam Olsen, aka Rhamphoryncus

On Tue, Sep 30, 2008 at 3:18 PM, Nick Coghlan wrote:

That said, I don't think this is something we (or, more to the point, Guido) need to make a decision on right now - for 3.0, having bytes-level APIs that can see everything, and Unicode APIs that ignore badly encoded filenames is worth trying. If it proves inadequate, then we can revisit the idea of some kind of implicit escaping mechanism in the Unicode APIs for 3.1 when there is more time for a proper PEP.

Right. Given that most syscalls already support both bytes and (unicode) str, the simplest thing to do is to take this a bit further, along the lines of Victor's patches, which I'm reviewing in Rietveld right now: http://codereview.appspot.com/3055 -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 10:50 am, eckhardt@satorlaser.com wrote:

On Sunday 28 September 2008, Gregory P. Smith wrote:

...

"broken" systems will always exist. Code to deal with them must be possible to write in python 3.0.

...

since any given path (not just fs) can have its own encoding it makes the most sense to me to let the OS deal with the errors and not try to enforce bytes vs string encoding type at the python lib. level.

Actually I'm afraid that that isn't really useful. I, too, would like to kick peoples' back in order to get the to fix their systems or use the proper codepage while mounting etc, etc, but that is not going to happen soon. Just ignoring those broken systems is tempting, but alienating a large group of users isn't IMHO worth it.

Instead, I'd like to present a different approach:

1. For POSIX platforms (using a byte string for the path): Here, the first approach is to convert the path to Unicode, according to the locale's CTYPE category. Hopefully, it will be UTF-8, but also codepages should work. If there is a segment (a byte sequence between two path separators) where it doesn't work, it uses an ASCII mapping where possible and codepoints from the "Private Use Area" (PUA) of Unicode for the non-decodable bytes.

In order to pass this path to fopen(), each segment would be converted to a byte string again, using the locale's CTYPE category except for segments which use the PUA where it simply encodes the original bytes.

That's a cool idea, but this encoding hack would need to be clearly documented and exposed for when you need to talk to another piece of software about pathnames. Consider a Python implementation of "xargs". Right now this can be implemented as a pretty simple for loop which eventually invokes 'subprocess.call' or similar. http://docs.python.org/dev/3.0/library/os.html#process-management doesn't say what the type of the arguments to the various 'exec' variants are - one presumes they'd have to be bytes. Not all arguments to subprocesses need to be filenames, but when they are they need to be encoded appropriately. Also, consider the following nightmare scenario: a system which has two users with incompatible locales. One wishes to write a "text" (ha ha) file with a list of pathnames in it to share with the other. What encoding should that file be in? How should the other user know how to interpret it? (And of course: what if that user is going to be piping that file to "xargs", or the original file came out of "find"?) I don't think that you can do encoding a segment at a time here, at least not at the API level; however, the whole file could be written in the py-posix- paths encoding which does exactly what you propose.

2. For win32 platforms, the path is already Unicode (UTF-16) and the whole problem is solved or not solved by the OS.

If the "or not solved" part of that is true then this probably bears further investigation. I suspect that the OS *always* provides some solution, even if it's the wrong solution, though. Also, what about MacOS X?

In the end, both approaches yield a path represented by a Unicode string for intermediate use, which provides maximum flexibility. Further, it preserves "broken" encodings by simply mapping their byte-values to the PUA of Unicode. Maybe not using a string to represent a path would be a good idea, too. At least it would make it very clear that the string is not completely free-form.

Personally, I plan to use this: http://twistedmatrix.com/documents/8.1.0/api/twisted.python.filepath.FilePat... for all of my file I/O in the future. For what it's worth, this object _doesn't_ handle unicode properly and it's been a thorn in our side for quite a while. We have plans to implement some kind of unicode-friendly API which is compatible with 2.6; if we have any brilliant ideas I'll let you know, but I doubt they'll be in time. The general idea right now is that we'll keep around the original bytes returned from filesystem inspection and provide some context-sensitive encoding/decoding APIs for different applications. The PUA approach would allow us to maintain an API compatible with that. I would not actually mind if there were a POSIX-specific module we had to use to get every arcane nuance of brokenness of writing pathnames into text files to be correct, since Windows needs to come up with _some_ valid unicode filename for every file in the system (even if it's improperly decoded).

On Mon, 29 Sep 2008 14:34:07 +0200, Ulrich Eckhardt wrote:

On Monday 29 September 2008, glyph@divmod.com wrote:

...

Also, what about MacOS X?

AFAIK, OS X guarantees UTF-8 for filesystem encodings. So the OS also provides Unicode filenames and how it deals with broken or legacy media is left up to the OS.

Read Jack Jansen's recent email about NFC vs NFD. Jean-Paul

Greg Ewing writes:

Ulrich Eckhardt wrote:

...

AFAIK, OS X guarantees UTF-8 for filesystem encodings. So the OS also provides Unicode filenames and how it deals with broken or legacy media is left up to the OS.

Does this mean that the OS always returns valid utf-8 strings from filesystem calls, even if the media is broken or legacy?

No, this means Ulrich is wrong. NFD-normalized UTF-8 is more or less enforced by the default filesystem, but Mac OS X up to 10.4 at least also supports the FreeBSD filesystems, and some of those can have any encoding you like or none at all (ie, KOI8-R and Shift JIS in the same directory is possible). If you have a Mac it's easy enough to test by creating a disk image with a non-default file system.

I'm a bit late to join in this discussion, but if unicode filenames are going to be the normal mode, how about this whole normalized/ canonical business? This is a headache that has shown up on the Mac a couple of times, because MacOS prefers filenames to be NFC, whereas Python prefers its Unicode to be NFD (or the other way around, I keep forgetting the details). To make the problem worse, even though MacOS prefers its filenames in the one form, it will allow filenames in the other form (which can happen if you mount a foreign filesystem, for example over the net). The fact that "incorrect" filenames can exist mean that the simple solution of converting NFC<->NFD in Python's open() and friends won't work (or, at least, it'll make some filenames inaccessible, and listdir() may return filenames that don't exist). -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman