Martin Blais <blais@furius.ca> writes:

What if we could completely disable the implicit conversions between unicode and str? In other words, if you would ALWAYS be forced to call either .encode() or .decode() to convert between one and the other... wouldn't that help a lot deal with that issue?

I don't know. I've made one or two apps safe against this and it's mostly just annoying.

How hard would that be to implement?

import sys reload(sys) sys.setdefaultencoding('undefined')

Would it break a lot of code? Would some people want that? (I know I would, at least for some of my code.) It seems to me that this would make the code more explicit and force the programmer to become more aware of those conversions. Any opinions welcome.

I'm not sure it's a sensible default. Cheers, mwh -- It is never worth a first class man's time to express a majority opinion. By definition, there are plenty of others to do that. -- G. H. Hardy

Le lundi 03 octobre 2005 à 02:09 -0400, Martin Blais a écrit :

What if we could completely disable the implicit conversions between unicode and str?

This would be very annoying when dealing with some modules or libraries where the type (str / unicode) returned by a function depends on the context, build, or platform. A good rule of thumb is to convert to unicode everything that is semantically textual, and to only use str for what is to be semantically treated as a string of bytes (network packets, identifiers...). This is also, AFAIU, the semantic model which is favoured for a hypothetical future version of Python. This is what I'm using to do safe conversion to a given type without worrying about the type of the argument: DEFAULT_CHARSET = 'utf-8' def safe_unicode(s, charset=None): """ Forced conversion of a string to unicode, does nothing if the argument is already an unicode object. This function is useful because the .decode method on an unicode object, instead of being a no-op, tries to do a double conversion back and forth (which often fails because 'ascii' is the default codec). """ if isinstance(s, str): return s.decode(charset or DEFAULT_CHARSET) else: return s def safe_str(s, charset=None): """ Forced conversion of an unicode to string, does nothing if the argument is already a plain str object. This function is useful because the .encode method on an str object, instead of being a no-op, tries to do a double conversion back and forth (which often fails because 'ascii' is the default codec). """ if isinstance(s, unicode): return s.encode(charset or DEFAULT_CHARSET) else: return s Good luck Antoine.

Le lundi 03 octobre 2005 à 14:59 +0200, Fredrik Lundh a écrit :

Antoine Pitrou wrote:

...

A good rule of thumb is to convert to unicode everything that is semantically textual

and isn't pure ASCII.

How can you be sure that something that is /semantically textual/ will always remain "pure ASCII" ? That's contradictory, unless your software never goes out of the anglo-saxon world (and even...).

(anyone who are tempted to argue otherwise should benchmark their applications, both speed- and memorywise, and be prepared to come up with very strong arguments for why python programs shouldn't be allowed to be fast and memory-efficient whenever they can...)

I think most applications don't critically depend on text processing performance. OTOH, international adaptability is the kind of thing that /will/ bite you one day if you don't prepare for it at the beginning. Also, if necessary, the distinction could be an implementation detail and the conversion be transparent (like int vs. long): the text would be coded in an 8-bit charset as long as possible and converted to a wide encoding only when necessary. The important thing is that these optimisations, if they are necessary, should be transparently handled by the Python runtime. (it seems to me - I may be mistaken - that modern Windows versions treat every string as 16-bit unicode internally. Why are they doing it if it is that inefficient?) Regards Antoine.

Antoine Pitrou wrote:

...

...

A good rule of thumb is to convert to unicode everything that is semantically textual

and isn't pure ASCII.

How can you be sure that something that is /semantically textual/ will always remain "pure ASCII" ?

"is" != "will always remain" </F>

M.-A. Lemburg wrote:

Michael Hudson wrote:

...

Martin Blais <blais@furius.ca> writes:

...

What if we could completely disable the implicit conversions between unicode and str? In other words, if you would ALWAYS be forced to call either .encode() or .decode() to convert between one and the other... wouldn't that help a lot deal with that issue?

I don't know. I've made one or two apps safe against this and it's mostly just annoying.

...

How hard would that be to implement?

import sys reload(sys) sys.setdefaultencoding('undefined')

You shouldn't post tricks like these :-)

The correct way to change the default encoding is by providing a sitecustomize.py module which then call the sys.setdefaultencoding("undefined").

This is a much more evil trick IMO, as it affects all Python code, rather than a single program. I would argue that it's evil to change the default encoding in the first place, except in this case to disable implicit encoding or decoding. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Antoine Pitrou <solipsis@pitrou.net> wrote:

Le lundi 03 octobre 2005 à 14:59 +0200, Fredrik Lundh a écrit :

...

Antoine Pitrou wrote:

...

A good rule of thumb is to convert to unicode everything that is semantically textual

and isn't pure ASCII.

How can you be sure that something that is /semantically textual/ will always remain "pure ASCII" ? That's contradictory, unless your software never goes out of the anglo-saxon world (and even...).

Non-unicode text input widgets. Works great. Can be had with the ANSI wxPython installation.

(it seems to me - I may be mistaken - that modern Windows versions treat every string as 16-bit unicode internally. Why are they doing it if it is that inefficient?)

Because modern Windows supports all sorts of symbols which are necessary for certain special English uses (greek symbols for math, etc.), and trying to have all of them without just using the unicode backend that is used for all of the international "builds" (isn't it just a language definition?) anyways, would be a waste of time/effort. - Josiah

Hi, Josiah:

...

How can you be sure that something that is /semantically textual/ will always remain "pure ASCII" ? That's contradictory, unless your software never goes out of the anglo-saxon world (and even...).

Non-unicode text input widgets.

You didn't understand my statement. I didn't mean : - how can you /technically enforce/ no unicode text at all but : - how can you be sure that your users will never /want/ to enter some text that can't be represented with the current 8-bit charset? Of course the answer to the latter is: you can't. Fredrik:

Under the default encoding (and quite a few other encodings), that's true for plain ascii strings and Unicode strings.

If I have an unicode string containing legal characters greater than 0x7F, and I pass it to a function which converts it to str, the conversion fails. If I have an 8-bit string containing legal non-ascii characters in it (for example the name of a file as returned by the filesystem, which I of course have no prior control on), and I give it to a function which does an implicit conversion to unicode, the conversion fails. Here is an example so that you really understand. I am under a French locale (iso-8859-15), let's just try to enter a French word and see what happens when converting to unicode: -> As a string constant:

...

...

s = "été" s '\xe9t\xe9' u = unicode(s) Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 0: ordinal not in range(128)

-> By asking for input:

...

...

s = raw_input() été s '\xe9t\xe9' unicode(s) Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 0: ordinal not in range(128)

It should work, but it fails miserably. In the current situation, if the programmer doesn't carefully plan for these cases by manually managing conversions (which of course he can do - but it's boring and bothersome - not to mention that many programmers do not even understand the issue!), some users will see the program die with a nasty exception, just because they happen to need a bit more than the plain latin alphabet without diacritics. (even the standard Python library is bitten: witness the weird getcwd() / getcwdu() pair...) I find it surprising that you claim there is no difficulty when everything points to the contrary. See for example how often confused developers ask for help on mailing-lists... Regards Antoine.

Hi, Le lundi 03 octobre 2005 à 20:37 +0200, Fredrik Lundh a écrit :

...

If I have an unicode string containing legal characters greater than 0x7F, and I pass it to a function which converts it to str, the conversion fails.

so? if it does that, it's not unicode safe. [...] what's that has to do with my argument (which is that you can safely mix ascii strings and unicode strings, because that's how things were designed).

If that's how things were designed, then Python's entire standard library (not to mention third-party libraries) is not "unicode safe" - to quote your own words - since many functions may return 8-bit strings containing non-ascii characters. There lies the problem for many people, until the stdlib is fixed - or until the string types are changed. That's why you very regularly see people complaining about how conversions sometimes break their code in various ways. Anyway, I don't think we will reach an agreement here. We have different expectations w.r.t. to how the programming language may/should handle general text. I propose we end the discussion. Regards Antoine.

...

If that's how things were designed, then Python's entire standard brary (not to mention third-party libraries) is not "unicode safe" - to quote your own words - since many functions may return 8-bit strings containing non-ascii characters.

huh? first you talk about functions that convert unicode strings to 8-bit strings, now you talk about functions that return raw 8-bit strings?

Are you deliberately missing the argument? And can't you understand that conversions are problematic in both directions (str -> unicode /and/ unicode -> str)? If an stdlib function returns an 8-bit string containing non-ascii data, then this string used in unicode context incurs an implicit conversion, which fails. How's that for "unicode safety" of stdlib functions? Will you argue that this gives no difficulties to anyone ?

all this in response to a post that argues that it's in fact a good idea to use plain strings to hold textual data that happens to contain ASCII only,

To which you apparently didn't read my answer, that is: you can never be sure that a variable containing something which is /semantically/ textual (*) will never contain anything other than ASCII text. For example raw_input() won't tell you that its 8-bit string result contains some chars > 0x7F. Same for many other library functions. How do you cope with (more or less occasional) non-ascii data coming in as 8-bit strings? (*) that is, contains some natural language Either you carefully plan for non-ascii text coming in your application (including workarounds against Python's ascii-by-default conversion policy), or you deliberately cripple your application by deciding that non-ASCII text is forbidden in (some or all) places. Choose the latter and you'll be hostile to users. And this thread began with a poster who found difficult the way implicit conversions happen in Python. So it's very funny that you deny the existence of a problem for certain developers. Antoine.

Presumably in Python 3.0, opening a file in "text" mode will require an encoding to be specified, and opening it in "binary" mode will cause it to produce or consume byte arrays, not strings. This should apply to sockets too, and really any I/O facility, including GUI frameworks, DBAPI objects, os.listdir(), etc.

Great :)

Of course, to get there we really need to add a convenient bytes type, perhaps by enhancing the current 'array' module. It'd be nice to have a way to get this in 2.x versions so people can start fixing stuff to work the right way.

Could the "bytes" type be just the same as the current "str" type but without the implicit unicode conversion ? Or am I missing some desired functionality ?

Antoine Pitrou wrote:

To which you apparently didn't read my answer, that is: you can never be sure that a variable containing something which is /semantically/ textual (*) will never contain anything other than ASCII text.

That is simply not true. There are variables that is semantically textual, yet I can be sure that this is a byte string only if it consists just of ASCII. For example, if you invoke a Tkinter function, it will return a byte string if the result is purely ASCII, else return a Unicode string. This is an interface guarantee, hence I can be sure. Regards, Martin

Antoine> If an stdlib function returns an 8-bit string containing Antoine> non-ascii data, then this string used in unicode context incurs Antoine> an implicit conversion, which fails. Such strings should be converted to Unicode at the point where they enter the application. That's likely the only place where you have a good chance of knowing the data encoding. Files generally have no encoding information associated with them. Some databases don't handle Unicode transparently. If you hang onto the input from such devices as plain strings until you need them as Unicode, you will almost certainly not know how the string was encoded. The state of the outside Unicode world being as miserable as it is (think web input forms), you often don't know the encoding at the interface and have to guess anyway. Even so, isolating that guesswork to the interface is better than recovering somewhere further downstream. Skip

Le lundi 03 octobre 2005 à 14:02 -0700, Guido van Rossum a écrit :

On 10/3/05, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

Could the "bytes" type be just the same as the current "str" type but without the implicit unicode conversion ? Or am I missing some desired functionality ?

No. It will be a mutable array of bytes. It will intentionally resemble strings as little as possible. There won't be a literal for it.

Thinking about it, it may have to offer the search and replace facilities offered by strings (including regular expressions). Here is an use case : say I'm reading an HTML file (or receiving it over the network). Since the character encoding can be specified in the HTML file itself (in the <head>...</head>), I must first receive it as a bytes object. But then I must fetch the encoding information from the HTML header: therefore I must use some string ops on the bytes object to parse this information. Only after I have discovered the encoding, can I finally convert the bytes object to a text string. Or would there be another way to do it?

Le lundi 03 octobre 2005 à 17:42 -0700, Guido van Rossum a écrit :

I don't see a use case for replace.

Agreed.

Alternatively, you could always specify Latin-1 as the encoding and convert it that way -- I don't think there's any input that can cause Latin-1 decoding to fail.

You seem to be right. « In 1992, the IANA registered the character map ISO-8859-1 (note the extra hyphen), a superset of ISO/IEC 8859-1, for use on the Internet. This map assigns control characters to the code values 00-1F, 7F, and 80-9F. It thus provides for 256 characters via every possible 8-bit value. » http://en.wikipedia.org/wiki/ISO_8859-1#ISO-8859-1 Regards Antoine.

James Y Knight wrote:

Your point would be much easier to stomach if the "str" type could *only* hold 7-bit ASCII.

why? strings are not mutable, so it's not like an ASCII string will suddenly sprout non-ASCII characters. what ends up in a string is defined by the string source. if you cannot trust the source, your programs will never work. after all, there's no- thing in Python that keeps things like: s = file.readline().decode("iso-8859-1") s = elem.findtext("node") s = device.read_encoded_data() from returning integers instead of strings, or returning socket objects on odd fridays. but if the interface spec says that they always return strings that adher to python's text model (=unicode or things that can be mixed with unicode), you can trust them as much as you can trust anything else in Python. (this is of course also why we talk about file-like objects in Python, and sequences, and iterators and iterables, and stuff like that. it's not type(obj) that's important, it's what you can do with obj and how it behaves when you do it) </F>

On 10/3/05, Michael Hudson <mwh@python.net> wrote:

Martin Blais <blais@furius.ca> writes:

...

How hard would that be to implement?

import sys reload(sys) sys.setdefaultencoding('undefined')

Hmmm any particular reason for the call to reload() here?

On 10/15/05, Reinhold Birkenfeld <reinhold-birkenfeld-nospam@wolke7.net> wrote:

Martin Blais wrote:

...

On 10/3/05, Michael Hudson <mwh@python.net> wrote:

...

Martin Blais <blais@furius.ca> writes:

...

How hard would that be to implement?

import sys reload(sys) sys.setdefaultencoding('undefined')

Hmmm any particular reason for the call to reload() here?

Yes. setdefaultencoding() is removed from sys by site.py. To get it again you must reload sys.

Thanks. cheers,