On 2008-01-10 14:31, Eric Smith wrote:

(I'm posting to python-dev, because this isn't strictly 3.0 related. Hopefully most people read it in addition to python-3000).

I'm working on backporting the changes I made for PEP 3101 (Advanced String Formatting) to the trunk, in order to meet the pre-PyCon release date for 2.6a1.

I have a few questions about how I should handle str/unicode. 3.0 was pretty easy, because everything was unicode.

Since this is a new feature, why bother with strings at all (even in 2.6) ?

Use Unicode throughout and be done with it.

1: How should the builtin format() work? It takes 2 parameters, an object o and a string s, and returns o.__format__(s). If s is None, it returns o.__format__(empty_string). In 3.0, the empty string is of course unicode. For 2.6, should I use u'' or ''?

2: In 3.0, object.__format__() is essentially this:

class object:
    def __format__(self, format_spec):
        return format(str(self), format_spec)

In 2.6, I assume it should be the equivalent of:

class object:
    def __format__(self, format_spec):
        if isinstance(format_spec, str):
            return format(str(self), format_spec)
        elif isinstance(format_spec, unicode):
            return format(unicode(self), format_spec)
        else:
            error

 Does that seem right?

3: Every overridden __format__() method is going to have to check for string or unicode, just like object.__format() does, and return either a string or unicode object, appropriately. I don't see any way around this, but I'd like to hear any thoughts. I guess there aren't all that many __format__ methods that will be implemented, so this might not be a big burden. I'll of course implement the built in ones.

Thanks in advance for any insights.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Jan 10, 2008, at 9:07 AM, M.-A. Lemburg wrote:

On 2008-01-10 14:31, Eric Smith wrote:

...

(I'm posting to python-dev, because this isn't strictly 3.0 related. Hopefully most people read it in addition to python-3000).

I'm working on backporting the changes I made for PEP 3101 (Advanced String Formatting) to the trunk, in order to meet the pre-PyCon release date for 2.6a1.

I have a few questions about how I should handle str/unicode. 3.0 was pretty easy, because everything was unicode.

Since this is a new feature, why bother with strings at all (even in 2.6) ?

Use Unicode throughout and be done with it.

+1 - -Barry

On Jan 10, 2008 9:57 AM, Eric Smith eric+python-dev@trueblade.com wrote:

Eric Smith wrote:

...

(I'm posting to python-dev, because this isn't strictly 3.0 related. Hopefully most people read it in addition to python-3000).

I'm working on backporting the changes I made for PEP 3101 (Advanced String Formatting) to the trunk, in order to meet the pre-PyCon release date for 2.6a1.

I have a few questions about how I should handle str/unicode. 3.0 was pretty easy, because everything was unicode.

1: How should the builtin format() work? It takes 2 parameters, an object o and a string s, and returns o.__format__(s). If s is None, it returns o.__format__(empty_string). In 3.0, the empty string is of course unicode. For 2.6, should I use u'' or ''?

I just re-read PEP 3101, and it doesn't mention this behavior with None. The way the code actually works is that the specifier is optional, and if it isn't present then it defaults to an empty string. This behavior isn't mentioned in the PEP, either.

This feature came from a request from Talin[0]. We should either add this to the PEP (and docs), or remove it. If we document it, it should mention the 2.x behavior (as other places in the PEP do). If we removed it, it would remove the one place in the backport that's not just hard, but ambiguous. I'd just as soon see the feature go away, myself.

IIUC, the 's' argument is the format specifier. Format specifiers are written in a very conservative character set, so I'm not sure it matters. Or are you assuming that the *type* of 's' also determines the type of the output?

I may be in the minority here, but I think I like having a default for 's' (as implemented -- the PEP ought to be updated) and I also think it should default to an 8-bit string, assuming you support 8-bit strings at all -- after all in 2.x 8-bit strings are the default string type (as reflected by their name, 'str').

...

3: Every overridden __format__() method is going to have to check for string or unicode, just like object.__format() does, and return either a string or unicode object, appropriately. I don't see any way around this, but I'd like to hear any thoughts. I guess there aren't all that many __format__ methods that will be implemented, so this might not be a big burden. I'll of course implement the built in ones.

The PEP actually mentions that this is how 2.x will have to work. So I'll go ahead and implement it that way, on the assumption that getting string support into 2.6 is desirable.

I think it is. (But then I still live in a predominantly ASCII world. :-)

For data types whose output uses only ASCII, would it be acceptable if they always returned an 8-bit string and left it up to the caller to convert it to Unicode? This would apply to all numeric types. (The date/time types have a strftime() style API which means the user must be able to specifiy Unicode.)

Guido van Rossum wrote:

For data types whose output uses only ASCII, would it be acceptable if they always returned an 8-bit string and left it up to the caller to convert it to Unicode? This would apply to all numeric types. (The date/time types have a strftime() style API which means the user must be able to specifiy Unicode.)

To elaborate on this a bit (and handwaving a lot of important details out of the way) do you mean something like the following for the builtin format?:

def format(obj, fmt_spec=None): if fmt_spec is None: fmt_spec='' result = obj.__format__(fmt_spec) if isinstance(fmt_spec, unicode): if isinstance(result, str): result = unicode(result) return result

Cheers, Nick.

Nick Coghlan wrote:

Guido van Rossum wrote:

...

For data types whose output uses only ASCII, would it be acceptable if they always returned an 8-bit string and left it up to the caller to convert it to Unicode? This would apply to all numeric types. (The date/time types have a strftime() style API which means the user must be able to specifiy Unicode.)

To elaborate on this a bit (and handwaving a lot of important details out of the way) do you mean something like the following for the builtin format?:

def format(obj, fmt_spec=None): if fmt_spec is None: fmt_spec='' result = obj.__format__(fmt_spec) if isinstance(fmt_spec, unicode): if isinstance(result, str): result = unicode(result) return result

Isn't unicode idempotent? Couldn't

if isinstance(result, str): result = unicode(result)

avoid repeating in Python a test already made in C by re-spelling it as

result = unicode(result)

or have you hand-waved away important details that mean the test really is required?

regards Steve

Eric Smith wrote:

...

Guido van Rossum wrote:

...

For data types whose output uses only ASCII, would it be acceptable if they always returned an 8-bit string and left it up to the caller to convert it to Unicode? This would apply to all numeric types. (The date/time types have a strftime() style API which means the user must be able to specifiy Unicode.)

I'm finally getting around to finishing this up. The approach I've taken for int, long, and float, is that they take either unicode or str format specifiers, and always return str results. The builtin format() deals with converting str to unicode, if the format specifier was originally unicode. This all works great. It allows me to easily implement both ''.format and u''.format taking int, long, and float parameters.

I'm now working on datetime. The __format__ method is really just a wrapper around strftime. I was assuming (or rather hoping) that strftime does the right thing with unicode and str (unicode in = unicode out, str in = str out). But it turns out strftime doesn't accept unicode:

$ ./python Python 2.6a0 (trunk:60845M, Feb 15 2008, 21:09:57) [GCC 4.1.2 20070626 (Red Hat 4.1.2-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

import datetime datetime.date.today().strftime('%y')

'08'

...

...

datetime.date.today().strftime(u'%y')

Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: strftime() argument 1 must be str, not unicode

As part of this task, I'm really not up to the job of changing strftime to support both str and unicode inputs. So I think I'll put all of the __format__ code in place to support it if and when strftime supports unicode. In the meantime, it won't be possible for u''.format to work with datetime objects.

...

...

'year: {0:%y}'.format(datetime.date.today())

'year: 08'

...

...

u'year: {0:%y}'.format(datetime.date.today())

Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: strftime() argument 1 must be str, not unicode

The bad error message is a result of __format__ passing on unicode to strftime.

There are, of course, various ugly ways to work around this involving nested format calls.

Maybe I'll extend strftime to unicode for the PyCon sprint.

Eric.

* Nick Coghlan wrote:

Eric Smith wrote:

...

The bad error message is a result of __format__ passing on unicode to strftime.

There are, of course, various ugly ways to work around this involving nested format calls.

I don't know if this fits your definition of "ugly workaround", but what if datetime.__format__ did something like:

def __format__(self, spec): encoding = None if isinstance(spec, unicode): encoding = 'utf-8' spec = spec.encode(encoding) result = strftime(spec, self) if encoding is not None: result = result.decode(encoding) return result

Note that hardcoding utf-8 is a bad guess here as strftime(3) emits locale strings, so decoding will easily fail.

I guess, a clean and complete solution (besides re-implementing the whole thing) would be to resolve each single format character with strftime, decode according to the locale and re-assemble the result string piece by piece. Doh!

nd

* Eric Smith wrote:

André Malo wrote:

...

I guess, a clean and complete solution (besides re-implementing the whole thing) would be to resolve each single format character with strftime, decode according to the locale and re-assemble the result string piece by piece. Doh!

That's along the lines of what I was thinking. strftime already does some of this to support %[zZ].

But now that I look at time.strftime in py3k, it's converting the entire unicode string to a char string with PyUnicode_AsString, then converting back with PyUnicode_Decode.

Looks wrong to me, too... :-)

nd