Barry Warsaw wrote:

Another decidedly ignorant Anglophone here, but one problem that I see with localizing stuff is that locale is app- (or at least thread-) global, isn't it? That would suck for applications like Mailman which are (going to be) multilingual in the sense that a single instance of the application will serve up documents in many languages, as opposed to serving up documents in just one of a choice of languages.

If it seems I don't know what I'm talking about, you're probably right. I just wanted to point out that there are applications have to deal with many languages at the same time.

Applications may also have to deal with output devices (i.e. GUI toolkits, printers, communication links) that don't necessarily have the same restrictions as the "default console". better do it the right way: deal with encodings at the boundaries, not inside the application. </F>

Ka-Ping Yee wrote:

On Fri, 19 May 2000, Guido van Rossum wrote:

...

The email below suggests a simple solution to a problem that e.g. Fran\347ois Pinard brought up long ago; repr() of a string turns all non-ASCII chars into \oct escapes. Jyrki's solution: use isprint(), which makes it locale-dependent. I can live with this.

Changing the behaviour of repr() (a function that internally converts data into data) based on a fixed global system parameter makes me uncomfortable. Wouldn't it make more sense for the locale business to be a property of the stream that the string is being printed on?

Umm, Jyrki's patch does *not* affect repr(): it's a patch to the string_print API which is used for the tp_print slot, so the only effect to be seen is when printing a string to a real file object (tp_print is only used by PyObject_Print() and that API is only used for writing to real PyFileObjects -- all other stream get the output of str() or repr()). Perhaps we should drop tp_print for strings altogether and let str() and repr() to decide what to do... (this is what Unicode objects do). The only good reason for implementing tp_print is to write huge amounts of data to a stream without creating intermediate objects -- not really needed for strings, since these *are* the intermediate object usually created for just this purpose ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Hi! Ka-Ping Yee:

On Fri, 19 May 2000, M.-A. Lemburg wrote:

...

Umm, Jyrki's patch does *not* affect repr(): it's a patch to the string_print API which is used for the tp_print slot,

Very sorry! I didn't actually look to see where the patch was being applied.

But then how can this have any effect on squishdot's indexing?

Sigh. Let me explain this in some detail. What do you see here: äöüÄÖÜß? If all went well, you should see some Umlauts which occur quite often in german words, like "Begrüssung", "ätzend" or "Grützkacke" and so on. During the late 80s we here Germany spend a lot of our free time to patch open source tools software like 'elm', 'B-News', 'less' and others to make them "8-Bit clean". For example on ancient Unices like SCO Xenix where the implementations of C-library functions like 'is_print', 'is_lower' where out of reach. After several years everybody seems to agree on ISO-8859-1 as the new european standard character set, which was also often losely called 8-Bit ASCII, because ASCII is a true subset of ISO latin1. Even at least the german versions of Windows used ISO-8859-1. As the WWW began to gain popularity nobody with a sane mind really used these splendid ASCII escapes like for example 'ä' instead of 'ä'. The same holds true for TeX users community where everybody was happy to type real umlauts instead of these ugly backslash escapes sequences used before: \"a\"o\"u ... To make a short: A lot of effort has been spend to make *ALL* programs 8-Bit clean: That is to move the bytes through without translating them from or into a bunch of incompatible multi bytes sequences, which nobody can read or even wants to look at. Now to get to back to your question: There are several nice HTML indexing engines out there. I personally use HTDig. At least on Linux these programs deal fine with HTML files containing 8-bit chars. But if for some reason Umlauts end up as octal escapes ('\344' instead of 'ä') due to the use of a Python 'print some_tuple' during the creation of HTML files, a search engine will be unable to find those words with escaped umlauts. Mit freundlichen Grüßen, Peter P.S.: Hope you didn't find my explanation boring or off-topic.

Ka-Ping Yee wrote:

On Fri, 19 May 2000, M.-A. Lemburg wrote:

...

Umm, Jyrki's patch does *not* affect repr(): it's a patch to the string_print API which is used for the tp_print slot,

Very sorry! I didn't actually look to see where the patch was being applied.

But then how can this have any effect on squishdot's indexing?

The only possible reason I can see is that this squishdot application uses 'print' to write the data -- perhaps it pipes it through some other tool ? -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Let's reboot this thread. Never mind the details of the actual patch, or why it would affect a particular index. Obviously if we're going to patch string_print() we're also going to patch string_repr() (and vice versa) -- the former (without the Py_PRINT_RAW flag) is supposed to be an optimization of the latter. (I hadn't even read the patch that far to realize that it only did one and not the other.) The point is simply this. The repr() function for a string turns it into a valid string literal. There's considerable freedom allowed in this conversion, some of which is taken (e.g. it prefers single quotes but will use double quotes when the string contains single quotes). For safety reasons, control characters are replaced by their octal escapes. This is also done for non-ASCI characters. Lots of people, most of them living in countries where Latin-1 (or another 8-bit ASCII superset) is in actual use, would prefer that non-ASCII characters would be left alone rather than changed into octal escapes. I think it's not unreasonable to ask that what they consider printable characters aren't treated as control characters. I think that using the locale to guide this is reasonable. If the locale is set to imply Latin-1, then we can assume that most output devices are capable of displaying those characters. What good does converting those characters to octal escapes do us then? If the input string was in fact binary goop, then the output will be unreadable goop -- but it won't screw up the output device (as control characters are wont to do, which is the main reason to turn them into octal escapes). So I don't see how the patch can do much harm, I don't expect that it will break much code, and I see a real value for those who use Latin-1 or other 8-bit supersets of ASCII. The one objection could be that the locale may be obsolescent -- but I've only heard /F vent an opinion about that; personally, I doubt that we will be able to remove the locale any time soon, even if we invent a better way. Plus, I think that "better way" should address this issue anyway. If the locale eventually disappears, the feature automatically disappears with it, because you *have* to make a locale.setlocale() call before the behavior of repr() changes. --Guido van Rossum (home page: http://www.python.org/~guido/)

Peter Funk wrote:

AFAIK locale and friends conform to POSIX.1. Calling this obsolescent... hmmm... may offend a *LOT* of people. Try this on comp.os.linux.advocacy ;-)

you're missing the point -- now that we've added unicode support to Python, the old 8-bit locale *ctype* stuff no longer works. while some platforms implement a wctype interface, it's not widely available, and it's not always unicode. so in order to provide platform-independent unicode support, Python 1.6 comes with unicode-aware and fully portable replacements for the ctype functions. the code is already in there...

On POSIX systems there are a several environment variables used to control the default locale settings for a users session. For example on my SuSE Linux system currently running in the german locale the environment variable LC_CTYPE=de_DE is automatically set by a file /etc/profile during login, which causes automatically the C-library function toupper('ä') to return an 'Ä' ---you should see a lower case a-umlaut as argument and an upper case umlaut as return value--- without having all applications to call 'setlocale' explicitly.

So this simply works well as intended without having to add calls to 'setlocale' to all application program using this C-library functions.

note that this leaves us with four string flavours in 1.6: - 8-bit binary arrays. may contain binary goop, or text in some strange encoding. upper, strip, etc should not be used. - 8-bit text strings using the system encoding. upper, strip, etc works as long as the locale is properly configured. - 8-bit unicode text strings. upper, strip, etc may work, as long as the system encoding is a subset of unicode -- which means US ASCII or ISO Latin 1. - wide unicode text strings. upper, strip, etc always works. is this complexity really worth it? </F>

Guido van Rossum wrote:

...

Peter Funk wrote:

...

AFAIK locale and friends conform to POSIX.1. Calling this obsolescent... hmmm... may offend a *LOT* of people. Try this on comp.os.linux.advocacy ;-)

you're missing the point -- now that we've added unicode support to Python, the old 8-bit locale *ctype* stuff no longer works. while some platforms implement a wctype interface, it's not widely available, and it's not always unicode.

Huh? We were talking strictly 8-bit strings here. The locale support hasn't changed there.

I meant that the locale support, even though it's part of POSIX, isn't good enough for unicode support...

...

so in order to provide platform-independent unicode support, Python 1.6 comes with unicode-aware and fully portable replacements for the ctype functions.

For those who only need Latin-1 or another 8-bit ASCII superset, the Unicode stuff is overkill.

why? besides, overkill or not:

...

the code is already in there...

...

note that this leaves us with four string flavours in 1.6:

- 8-bit binary arrays. may contain binary goop, or text in some strange encoding. upper, strip, etc should not be used.

These are not strings.

depends on who you're asking, of course:

...

...

b = fetch_binary_goop() type(b) dir(b) ['capitalize', 'center', 'count', 'endswith', 'expandtabs', ...

...

- 8-bit text strings using the system encoding. upper, strip, etc works as long as the locale is properly configured.

- 8-bit unicode text strings. upper, strip, etc may work, as long as the system encoding is a subset of unicode -- which means US ASCII or ISO Latin 1.

This is a figment of your imagination. You can use 8-bit text strings to contain Latin-1, but you have to set your locale to match.

if that's a supported feature (instead of being deprecated in favour for unicode), maybe we should base the default unicode/string con- versions on the locale too? background: until now, I've been convinced that the goal should be to have two "string-like" types: binary arrays for binary goop (including encoded text), and a Unicode-based string type for text. afaik, that's the solution used in Tcl and Perl, and it's also "conceptually compatible" with things like Java, Windows NT, and XML (and everything else from the web universe). given that, it has been clear to me that anything that is not compatible with this model should be removed as soon as possible (and deprecated as soon as we understand why it won't fly under the new scheme). but if backwards compatibility is more important than a minimalistic design, maybe we need three different "string-like" types: -- binary arrays (still implemented by the 8-bit string type in 1.6) -- 8-bit old-style strings (using the "system encoding", as defined by the locale. if the locale is not set, they're assumed to contain ASCII) -- unicode strings (possibly using a "polymorphic" internal representation) this also solves the default conversion problem: use the locale environ- ment variables to determine the default encoding, and call sys.set_string_encoding from site.py (see my earlier post for details). what have I missed this time? </F> PS. shouldn't sys.set_string_encoding be sys.setstringencoding?

...

...

sys ... 'set_string_encoding', 'setcheckinterval', 'setprofile', 'settrace', ...

looks a little strange...

Fredrik Lundh wrote:

...

...

- 8-bit text strings using the system encoding. upper, strip, etc works as long as the locale is properly configured.

- 8-bit unicode text strings. upper, strip, etc may work, as long as the system encoding is a subset of unicode -- which means US ASCII or ISO Latin 1.

This is a figment of your imagination. You can use 8-bit text strings to contain Latin-1, but you have to set your locale to match.

if that's a supported feature (instead of being deprecated in favour for unicode), maybe we should base the default unicode/string con- versions on the locale too?

This was proposed by Guido some time ago... the discussion ended with the problem of extracting the encoding definition from the locale names. There are some ways to solve this problem (static mappings, fancy LANG variables etc.), but AFAIK, there is no widely used standard on this yet, so in the end you're stuck with defining the encoding by hand... e.g. setenv LANG de_DE:latin-1 Perhaps we should help out a little and provide Python with a parser for the LANG variable with some added magic to provide useful defaults ?!

[...]

this also solves the default conversion problem: use the locale environ- ment variables to determine the default encoding, and call sys.set_string_encoding from site.py (see my earlier post for details).

Right, that would indeed open up a path for consent...

</F>

PS. shouldn't sys.set_string_encoding be sys.setstringencoding?

Perhaps... these were really only added as experimental feature to test the various possibilities (and a possible implementation). My original intention was removing these after final consent -- perhaps we should keep the functionality (expanded to a per thread setting; the global is a temporary hack) ?!

...

...

...

sys ... 'set_string_encoding', 'setcheckinterval', 'setprofile', 'settrace', ...

looks a little strange...

True; see above for the reason why ;-) PS: What do you think about the current internal design of sys.set_string_encoding() ? Note that hash() and the "st" parser markers still use UTF-8. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

On Mon, 22 May 2000, Ka-Ping Yee wrote:

I would like it to be only the latter, as Fred, i, and others

Please refer to Fredrik as Fredrik or /F; I don't think anyone else refers to him as "Fred", and I got really confused when I saw this! ;) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

Which is why I find Fredrik's attitude unproductive.

given that locale support isn't included if you make a default build, I don't think deprecating it would hurt that many people... but that's me; when designing libraries, I've always strived to find the *minimal* set of functions (and code) that makes it possible for a programmer to do her job well. I'm especially wary of blind alleys (sure, you can use locale, but that'll only take you this far, and you have to start all over if you want to do it right). btw, talking about productivity, go check out the case sensitivity threads on comp.lang.python. imagine if all those people hammered away on the 1.6 alpha instead...

And where's the SRE release?

at the usual place: http://w1.132.telia.com/~u13208596/sre/index.htm still one showstopper left, which is why I haven't made the long- awaited public "now it's finished, dammit" announcement yet. but it shouldn't be that far away. </F>

Hi! [...] [me]:

...

So this simply works well as intended without having to add calls to 'setlocale' to all application program using this C-library functions.

[Guido van Rossum]:

I don;t believe that. According to the ANSI standard, a C program *must* call setlocale(LC_..., "") if it wants the environment variables to be honored; without this call, the locale is always the "C" locale, which should *not* honor the environment variables.

pf@pefunbk> python Python 1.5.2 (#1, Jul 23 1999, 06:38:16) [GCC egcs-2.91.66 19990314/Linux (egcs- on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam

...

...

import string print string.upper("ä") Ä

This was the vanilla Python 1.5.2 as originally delivered by SuSE Linux. But yes, you are right. :-( My memory was confused by this practical experience. Now I like to quote from the man pages here: man toupper: [...] BUGS The details of what constitutes an uppercase or lowercase letter depend on the current locale. For example, the default "C" locale does not know about umlauts, so no con­ version is done for them. In some non - English locales, there are lowercase letters with no corresponding uppercase equivalent; the German sharp s is one example. man setlocale: [...] A program may be made portable to all locales by calling setlocale(LC_ALL, "" ) after program initialization, by using the values returned from a localeconv() call for locale - dependent information and by using strcoll() or strxfrm() to compare strings. [...] CONFORMING TO ANSI C, POSIX.1 Linux (that is, libc) supports the portable locales "C" and "POSIX". In the good old days there used to be sup­ port for the European Latin-1 "ISO-8859-1" locale (e.g. in libc-4.5.21 and libc-4.6.27), and the Russian "KOI-8" (more precisely, "koi-8r") locale (e.g. in libc-4.6.27), so that having an environment variable LC_CTYPE=ISO-8859-1 sufficed to make isprint() return the right answer. These days non-English speaking Europeans have to work a bit harder, and must install actual locale files. [...] In recent Linux distributions almost every Linux C-program seems to contain this obligatory 'setlocale(LC_ALL, "");' line, so it's easy to forget about it. However the core Python interpreter does not. it seems the Linux C-Library is not fully ANSI compliant in this case. It seems to honour the setting of $LANG regardless whether a program calls 'setlocale' or not. Regards, Peter

Hi, Guido van Rossum:

...

pf@pefunbk> python Python 1.5.2 (#1, Jul 23 1999, 06:38:16) [GCC egcs-2.91.66 19990314/Linux (egcs- on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam

...

...

...

import string print string.upper("ä") Ä

This threw me off too. However try this:

python -c 'print "ä".upper()'

Yes, you are right. :-( Conclusion: If the 'locale' module would ever become depreciated then ...ummm... we poor mortals will simply have to add a line 'import readline' to our Python programs. Nifty... ;-) Regards, Peter