Skip:

How is this exposed (if at all) to Python programmers?

Currently not exposed AFAICT except through calldll.

I happen to be developing on Linux, but the eventual delivery platform will be Windows. Is there no way to handle this in a cross-platform way?

Cross-platform is tricky as the file systems used on Linux have narrow string file names. Some higher level software (such as the forthcoming version of GTK+/GNOME) assume file names are encoded in UTF-8 but this is a somewhat dangerous assumption. The problem on Windows is that there are files you can not open by performing encoding operations on the Unicode names. They do have narrow generated names, but these are mangled and look like Z8F22~1.HTM so are hard to discover. Neil

On Windows NT/2K/XP the right thing to do is to use the wide char open function such as _CRTIMP FILE * __cdecl _wfopen(const wchar_t *, const wchar_t *); _CRTIMP int __cdecl _wopen(const wchar_t *, int, ...);

I agree. However: - Mark decided to take a different route, using fopen all the time, but encoding Unicode strings with the "mbcs" encoding, which calls MultiByteToWideCharCP with CP_ACP. AFAICT, this is correct as well (although it invokes an unneeded conversion of the string, since fopen, eventually, will convert the string back to Unicode - probably inside CreateFileExA - atleast on WinNT). In any case, passing Unicode objects to open() works just fine, atleast as long as they can be encoded in the ANSI code page. If you want to open a Chinese file name on a Russian Windows installation, you lose. - Skip was likely asking about a Unix installation, in which case all of this is irrelevant.

There may also be techniques for doing this on Windows 9x as the file system stores Unicode file names but I have never looked into this.

To my knowledge, VFAT32 doesn't - only NTFS does (which is not available on W9x). Regards, Martin

I want to be able to open all files on my English W2K install and can with many applications even if some have Chinese names and some have Russian. The big advance W2K made over NT was to only have one real version of the OS instead of multiple language versions.

I understand all that, but I can't agree with all your conclusions.

Locales are a really poor choice for people who need to operate in multiple languages and much software is moving to allowing concurrent use of multiple languages through the use of Unicode.

On Windows, locales and Unicode don't contradict each other. You can create files through the locale's code page, and they still end up on disk correctly. This is a much better situation than you have on Unix. In any case, there is no alternative. Locales may be good or bad - you must follow system conventions, if you want to write usable software.

...

To my knowledge, VFAT32 doesn't - only NTFS does (which is not available on W9x).

I have a file called u"C:\\z\u0439\u0446.html" on my W2K FAT partition which displays correctly in the explorer and can be opened in, for example, notepad.

Oops, you are right - the long file name is in Unicode. It is only when you do not have a long file name that the short one is interpreted in OEM encoding.

...

...

...

import glob glob.glob("C:\\*.html") ['C:\\l2.html', 'C:\\list.html', 'C:\\m4.html', 'C:\\x.html', 'C:\\z??.html'] for i in glob.glob("C:\\*.html"): ... f = open(i) ... Traceback (most recent call last): File "<stdin>", line 2, in ? IOError: [Errno 22] Invalid argument: 'C:\\z??.html'

I agree this is unfortunate; patches are welcome. Please notice that the strategy of using wchar_t API on Windows has explicitly been considered and rejected, for the complexity of the code changes involved. So anybody proposing a patch would need to make it both useful, and easy to maintain. With these constraints, the current implementation is the best thing Mark could come up with. Software always has limitations, which are removed only if somebody is bothered so much as to change the software. Regards, Martin

The system provided scripting languages support wide character file names.

Please understand that Python also supports wide character file names. It just doesn't allow all the possible values that the system would allow.

For Each f1 in fso.GetFolder("C:\").Files

That, of course, is another important difference: Here you get the directory contents as wide strings. Changing os.listdir to return Unicode objects would be possible, but would likely introduce a number of incompatibilities. Your script (e.g. the Python variant) is prepared that .Files returns Unicode objects. Making the same change in Python on all functions that return file names (i.e. listdir, glob, etc) is difficult - most likely, you'll have to make the return type a choice of the application. Regards, Martin

The only option I see to make this situation less painful is to write a filename subsystem which implements two generic APIs:

1. file open using strings and Unicode

I think this "pretty much" works in Python 2.2 already. It uses the "mbcs" encoding on Windows, and the locale's encoding on Unix if locale.setlocale has been called (and the C library is good enough). That might be still wrong if the file system expects UTF-8, or a fixed encoding (e.g. on an NTFS or VFAT partition mounted on Linux), but I don't think there is anything that can be done about this: It would be a misconfigured system if then the user doesn't also use an UTF-8 locale.

2. file listing using either Unicode or strings with a predefined encoding in the output list

That is something that certainly needs to be done. Having a PEP on that would be useful. Regards, Martin

We'd still need to support other OSes as well, though, and I don't think that putting all this code into fileobject.c is a good idea -- after all opening files is needed by some other parts of Python as well and may also be useful for extensions.

The stuff isn't in fileobject.c. Py_FileSystemDefaultEncoding is defined in bltinmodule.c. Also, on other OSes: You can pass Unicode object to open on all systems. If Py_FileSystemDefaultEncoding is NULL, it will fall back to site.encoding. Of course, if the system has an open function that expects wchar_t*, we might want to use that instead of going through a codec. Off hand, Win32 seems to be the only system where this might work, and even there, it won't work on Win95.

I'd suggest to implement something similiar to the DLL loading code which is also implemented as subsystem in Python.

I'd say this is over-designed. It is not that there are ten alternative approaches to doing encodings in file names, and we only support two of them, but it is rather that there are only two, and we support all three of them :-) Also, it is more difficult than threads: for threads, there is a fixed set of API features that need to be represented. Doing Py_UNICODE* opening alone is easy, but look at the number of posixmodule functions that all expect file names of some sort. Regards, Martin

That's the global, sure but the code using it is scattered across fileobject.c and the posix module. I think it would be a good idea to put all this file naming code into some Python/fileapi.c file which then also provides C APIs for extensions to use. These APIs should then take the file name as PyObject* rather than char* to enable them to handle Unicode directly.

What do you gain by that? Most of the posixmodule functions that take filenames are direct wrappers around the system call. Using another level of indirection is only useful if the fileapi.c functions are used in different places. Notice that each function (open, access, stat, etc) is used exactly *once* currently, so putting this all into a single place just makes the code more complex. The extensions module argument is a red herring: I don't think there are many extension modules out there which want to call access(2) but would like to do so using a PyObject* as the first argument, but numbers as the other arguments.

...

Of course, if the system has an open function that expects wchar_t*, we might want to use that instead of going through a codec. Off hand, Win32 seems to be the only system where this might work, and even there, it won't work on Win95.

I expect this to become a standard in the next few years.

I doubt that. Posix people (including developers of various posixish systems) have frequently rejected that idea in recent years. Even for the most recent system in this respect (OS X), we hear that they still open files with a char*, where char is byte - the only advancement is that there is a guarantee that those bytes are UTF-8. It turns out that this is all you need: with that guarantee, there is no need for an additional set of APIs. UTF-8 was originally invented precisely to represent file names (and was called UTF-1 at that time); it is more likely that more systems will follow this convention. If so, a global per-system file system encoding is all that's needed. The only problem is that on Windows, MS has already decided that the APIs are in CP_ANSI, so they cannot change it to UTF-8 now; that's why Windows will need special casing if people are unhappy with the "mbcs" approach (which some apparantly are).

...

Also, it is more difficult than threads: for threads, there is a fixed set of API features that need to be represented. Doing Py_UNICODE* opening alone is easy, but look at the number of posixmodule functions that all expect file names of some sort.

Doesn't that support the idea of having a small subsystem in Python which exposes the Unicode aware APIs to Python and its extensions ?

No. It is a lot of work, and an additional layer of indirection, with no apparent advantage. Feel free to write a PEP, though. Regards, Martin

This change works for me on Windows 2000 and allows access to all files no matter what the current code page is set to. On Windows 9x (not yet tested), the _wfopen call should fail causing a fallback to fopen. Possibly the OS should be detected instead and _wfopen not attempted on 9x.

Now that you have that change, please try to extend it to posixmodule.c. This is where I gave up. Notice that, with changing Py_FileSystemDefaultEncoding and open() alone, you have worsened the situation: os.stat will now fail on files with non-ASCII names on which it works under the mbcs encoding, because windows won't find the file (correct me if I'm wrong).

On 9x, mbcs may be a better choice of encoding although it may also be possible to ask the file system to find the wide character file name and return the mangled short name that can then be used by fopen.

It is not just 9x: if you have ten (*) different APIs to open a file, 10 different APIs to stat a file, and so on, and have to select some of them at compile time, and some of them at run-time, it gets messy very quickly. (*) I'd expect that other systems may also have proprietary system calls to do these things, using either wchar_t* or a proprietary Unicode type.

The best approach to me seems to be to make Py_FileSystemDefaultEncoding settable by the user, at least allowing the choice between 'utf-8' and 'mbcs' with a default of 'utf-8' on NT and 'mbcs' on 9x.

By the user, or by the application? How can the application make a more educated guess than Python proper? Alternatively, how can the user (or her Administrator) know what value to put in there? On Windows, probably neither is a good idea; if the file system default encoding is used in the future, fixing it at mbcs is the best I can think of.

Please criticise any stylistic or correctness issues in the code as it is my first modification to the Python sources.

The code looks fine. I'd encourage you to continue on that topic; just expect that it will need many more rounds for completion. Regards, Martin

...

Now that you have that change, please try to extend it to posixmodule.c. This is where I gave up.

OK. os.open, os.stat, and os.listdir now work. Placed temporarily at http://pythoncard.sourceforge.net/posixmodule.c

Looks good. The posix_do_stat changes contain an error; you have put Python API calls inside the BEGIN_ALLOW_THREADS block. That is wrong: you must always hold the interpreter lock when calling Python API. Also, when calling _wstati64, you might want to assert that the function pointer is _stati64. Likewise, the code inside posix_open should hold the interpreter lock.

os.listdir returns Unicode objects rather than strings. This makes glob.glob work as well so my earlier script that finds the *.html files and opens them works. Unfortunately, I expect most callers of glob() will be expecting narrow strings.

That is not that much of a problem; we could try to define API where it is the caller's choice. However, the size of your changes is really disturbing here. There used to be already four versions of listing a directory; now you've added a fifth one. And it isn't even clear whether this code works on W9x, is it? There must be a way to fold the different Windows versions into a single one; perhaps it is acceptable to drop Win16 support. I think three different versions should be offered to the end user: - path is plain string, result is list of plain strings - path is Unicode string, result is list of Unicode strings - path is Unicode string, result is list of plain strings Perhaps one could argue that the third version isn't really needed: anybody passing Unicode strings to listdir should be expected to get them back also. That would leave us with two functional features on windows. I envision a fragment that looks like this #ifdef windows if (argument is unicode string) { #define strings wide #include "listdir_win.h" #undef strings } else { convert argument to string #define strings narrow #include "listdir_win.h" #undef strings #endif If you provide a similar listdir_posix and listdir_os2, it should be possible to get a uniform implementation.

...

Notice that, with changing Py_FileSystemDefaultEncoding and open() alone, you have worsened the situation: os.stat will now fail on files with non-ASCII names on which it works under the mbcs encoding, because windows won't find the file (correct me if I'm wrong).

If you give it a file name encoded in the current code page then it may fail where it did not before.

I was actually talking about stat as a function that you haven't touched, yet. Now, os.rename will fail if you pass two Unicode strings referring to non-ASCII file names. posix_1str and posix_2str are like the stat implementation, except that you cannot know statically what the function pointer is. Regards, Martin

Neil Hodgson wrote:

Marc-Andre Lemburg:

...

The only option I see to make this situation less painful is to write a filename subsystem which implements two generic APIs:

1. file open using strings and Unicode

2. file listing using either Unicode or strings with a predefined encoding in the output list

I started work on this in C++ for my SciTE editor a couple of months ago but the design started to include stuff like 'are these two paths pointing at one file', converting between OpenVMS and Unix paths, and handling URLs (at least using ftp and http). My brain threatened to explode if it got any more complex so it got moved to the 'future niceness' pile.

I believe that we could do well with the following assumptions: a) strings passed to open() use whatever encoding is needed by the file system b) Unicode passed to open() are converted to whatever the file system needs by then open() API. This doesn't cover all the possibilities, but goes a long way. Joining paths between file systems should really be left to the os.path APIs. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

...

...

...

...

"Martin" == Martin v Loewis writes:

>> What's the correct way to deal with filenames in a Unicode >> environment? Consider this: >> >> >>> import site site.encoding >> 'latin-1' Martin> Setting site.encoding is certainly the wrong thing to do. How Martin> can you know all users of your system use latin-1? Why is setting site.encoding appropriate to your environment at the time you install Python wrong? I can't know that all users of my system (whatever the definition of "my system" is) will use latin-1. Somewhere along the way I have to make some assumptions, however. On any given computer I assume the people who install Python will set site.encoding appropriate to their environment. The example I used was latin-1 simply because the folks I'm working with are in Austria and they came up with the example. I assume the best default encoding for them is latin-1. The application writers themselves will have no problem restricting internal filenames to be ascii. I assume it users want to save files of their own, they will choose characters from the Unicode character set they use most frequently. So, my example used latin-1. I could just as easily have chosen something else. Martin> On my system, the following works fine Martin> >>> import locale ; locale.setlocale(locale.LC_ALL,"") Martin> 'LC_CTYPE=de_DE;LC_NUMERIC=de_DE;LC_TIME=de_DE;LC_COLLATE=C;LC_MONETARY=de_DE;LC_MESSAGES=de_DE;LC_PAPER=de_DE;LC_NAME=de_DE;LC_ADDRESS=de_DE;LC_TELEPHONE=de_DE;LC_MEASUREMENT=de_DE;LC_IDENTIFICATION=de_DE' Martin> >>> a = "abc\xe4\xfc\xdf.txt" u = unicode (a, "latin-1") open(u, "w") Martin> Martin> On Unix, your best bet for file names is to trust the user's Martin> locale settings. If you do that, open will accept Unicode Martin> objects. Martin> What is your locale? The above setlocale call prints 'LC_CTYPE=en_US;LC_NUMERIC=en_US;LC_TIME=en_US;LC_COLLATE=en_US;LC_MONETARY=en_US;LC_MESSAGES=en_US;LC_PAPER=en;LC_NAME=en;LC_ADDRESS=en;LC_TELEPHONE=en;LC_MEASUREMENT=en;LC_IDENTIFICATION=en' I can't get to the machines in Austria right now to see how their locales are set, though I suspect they haven't fiddled their LC_* environment, because they are having the problems I described. >> Is that the correct approach? Apparently Python's file object >> doesn't do this under the covers. Should it? Martin> No. There is no established convention, on Unix, how to do Martin> non-ASCII file names. If anything, following the user's locale Martin> setting is the most reasonable thing to do; this should be in Martin> synch of how the user's terminal displays characters. The Python Martin> installations' default encoding is almost useless, and shouldn't Martin> be changed. Martin> On Windows, things are much better, since there a notion of Martin> Unicode file names in the system. This suggests to me that the Python docs need some introductory material on this topic. It appears to me that there are two people in the Python community who live and breathe this stuff are you, Martin, and Marc-André. For most of the rest of us, especially if we've never conciously written code for consumption outside an ascii environment, the whole thing just looks like a quagmire. Skip