On 09/11/18 15:23, Victor Stinner wrote:

Hi,

Last week, I opened an issue to propose to add a new %T formatter to PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat() and PyErr_Format():

https://bugs.python.org/issue34595

I merged my change, but then Serhiy Storchaka asked if we can add something to get the "fully qualified name" (FQN) of a type, ex "datetime.timedelta" (FQN) vs "timedelta" (what I call "short" name). I proposed a second pull request to add %t (short) in addition to %T (FQN).

But then Petr Viktorin asked me to open a thread on python-dev to get a wider discussion. So here I am.

The rationale for this change is to fix multiple issues:

* C extensions use Py_TYPE(obj)->tp_name which returns a fully qualified name for C types, but the name (without the module) for Python name. Python modules use type(obj).__name__ which always return the short name.

That might be a genuine problem, but I wonder if "%T" is fixing the symptom rather than the cause here. Or is this only an issue for PyUnicode_FromFormat()?

* currently, many C extensions truncate the type name: use "%.80s" instead of "%s" to format a type name

That's an orthogonal issue -- you can change "%.80s" to "%s", and presumably you could use "%.80t" as well.

* "%s" with Py_TYPE(obj)->tp_name is used more than 200 times in the C code, and I dislike this complex pattern. IMHO "%t" with obj would be simpler to read, write and maintain.

I consider `Py_TYPE(obj)->tp_name` much more understandable than "%t". It's longer to spell out, but it's quite self-documenting.

* I want C extensions and Python modules to have the same behavior: respect the PEP 399. Petr considers that error messages are not part of the PEP 399, but the issue is wider than only error messages.

The other major use is for __repr__, which AFAIK we also don't guarantee to be stable, so I don't think PEP 399 applies to it. Having the same behavior between C and Python versions of a module is nice, but PEP 399 doesn't prescribe it. There are other differences as well -- for example, `_datetime.datetime` is immutable, and that's OK. If error messages and __repr__s should be consistent between Python and the C accelerator, are you planning to write tests for all the affected modules when switching them to %T/%t?

The main issue is that at the C level, Py_TYPE(obj)->tp_name is "usually" the fully qualified name for types defined in C, but it's only the "short" name for types defined in Python.

For example, if you get the C accelerator "_datetime", PyTYPE(obj)->tp_name of a datetime.timedelta object gives you "datetime.timedelta", but if you don't have the accelerator, tp_name is just "timedelta".

Another example, this script displays "mytimedelta(0)" if you have the C accelerator, but "__main__.mytimedelta(0)" if you use the Python implementation: --- import sys #sys.modules['_datetime'] = None import datetime

class mytimedelta(datetime.timedelta): pass

print(repr(mytimedelta())) ---

So I would like to fix this kind of issue.

Type names are mainly used for two purposes:

* format an error message * obj.__repr__()

It's unclear to me if we should use the "short" or the "fully qualified" name. It should maybe be decided on a case by case basis.

There is also a 3rd usage: to implement __reduce__, here backward compatibility matters.

Note: The discussion evolved since my first implementation of %T which just used the not well defined Py_TYPE(obj)->tp_name.

--

Petr asked me why not exposing functions to get these names. For example, with my second PR (not merged), there are 3 (private) functions:

/* type.__name__ */ const char* _PyType_Name(PyTypeObject *type); /* type.__qualname__ */ PyObject* _PyType_QualName(PyTypeObject *type); * type.__module__ "." type.__qualname__ (but type.__qualname__ for builtin types) */ PyObject * _PyType_FullName(PyTypeObject *type);

My concern here is that each caller has to handler error:

PyErr_Format(PyExc_TypeError, "must be str, not %.100s", Py_TYPE(obj)->tp_name);

would become:

PyObject *type_name = _PyType_FullName(Py_TYPE(obj)); if (name == NULL) { /* do something with this error ... */ PyErr_Format(PyExc_TypeError, "must be str, not %U", type_name); Py_DECREF(name);

When I report an error, I dislike having to handle *new* errors... I prefer that the error handling is done inside PyErr_Format() for me, to reduce the risk of additional bugs.

--

Serhiy also asked if we could expose the same feature at the *Python* level: provide something to get the fully qualified name of a type. It's not just f"{type(obj).__module}.{type(obj).__name__}", but you have to skip the module for builtin types like "str" (not return "builtins.str").

Maybe we can have "name: {0:t}, FQN: {0:T}".format(type(obj)). "t" for name and "T" for fully qualfied name. We would only have to modify type.__format__().

I'm not sure if we need to add new formatters to str % args.

Example of Python code:

raise TypeError("must be str, not %s" % type(fmt).__name__)

I'm not sure about Python changes. My first concern was just to avoid Py_TYPE(obj)->tp_name at the C level. But again, we should keep C and Python consistent. If the behavior of C extensions change, Python modules should be adapted as well, to get the same behavior.

Note: I reverted my change which added the %T formatter from PyUnicode_FromFormatV() to clarify the status of this issue.

Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com

FWIW, I personally think this went to python-dev prematurely. This is a shallow problem and IMO doesn't need all that much handwringing. Yes, there are a few different alternatives. So list them all concisely in the tracker and think about it for a few minutes and then pick one. No matter what's picked it'll be better than the status quo -- which is that printing a class name produces either a full or a short name depending on whether it's defined in C or Python, and there's no simple pattern (in C) to print either the full or the short name. On Tue, Sep 11, 2018 at 4:09 PM MRAB wrote:

On 2018-09-11 23:23, Victor Stinner wrote:

...

Hi,

Last week, I opened an issue to propose to add a new %T formatter to PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat() and PyErr_Format():

https://bugs.python.org/issue34595

I merged my change, but then Serhiy Storchaka asked if we can add something to get the "fully qualified name" (FQN) of a type, ex "datetime.timedelta" (FQN) vs "timedelta" (what I call "short" name). I proposed a second pull request to add %t (short) in addition to %T (FQN).

But then Petr Viktorin asked me to open a thread on python-dev to get a wider discussion. So here I am.

The rationale for this change is to fix multiple issues:

* C extensions use Py_TYPE(obj)->tp_name which returns a fully qualified name for C types, but the name (without the module) for Python name. Python modules use type(obj).__name__ which always return the short name.

* currently, many C extensions truncate the type name: use "%.80s" instead of "%s" to format a type name

* "%s" with Py_TYPE(obj)->tp_name is used more than 200 times in the C code, and I dislike this complex pattern. IMHO "%t" with obj would be simpler to read, write and maintain.

* I want C extensions and Python modules to have the same behavior: respect the PEP 399. Petr considers that error messages are not part of the PEP 399, but the issue is wider than only error messages.

The main issue is that at the C level, Py_TYPE(obj)->tp_name is "usually" the fully qualified name for types defined in C, but it's only the "short" name for types defined in Python.

For example, if you get the C accelerator "_datetime", PyTYPE(obj)->tp_name of a datetime.timedelta object gives you "datetime.timedelta", but if you don't have the accelerator, tp_name is just "timedelta".

Another example, this script displays "mytimedelta(0)" if you have the C accelerator, but "__main__.mytimedelta(0)" if you use the Python implementation: --- import sys #sys.modules['_datetime'] = None import datetime

class mytimedelta(datetime.timedelta): pass

print(repr(mytimedelta())) ---

So I would like to fix this kind of issue.

Type names are mainly used for two purposes:

* format an error message * obj.__repr__()

It's unclear to me if we should use the "short" or the "fully qualified" name. It should maybe be decided on a case by case basis.

There is also a 3rd usage: to implement __reduce__, here backward compatibility matters.

Note: The discussion evolved since my first implementation of %T which just used the not well defined Py_TYPE(obj)->tp_name.

--

Petr asked me why not exposing functions to get these names. For example, with my second PR (not merged), there are 3 (private) functions:

/* type.__name__ */ const char* _PyType_Name(PyTypeObject *type); /* type.__qualname__ */ PyObject* _PyType_QualName(PyTypeObject *type); * type.__module__ "." type.__qualname__ (but type.__qualname__ for builtin types) */ PyObject * _PyType_FullName(PyTypeObject *type);

My concern here is that each caller has to handler error:

PyErr_Format(PyExc_TypeError, "must be str, not %.100s", Py_TYPE(obj)->tp_name);

would become:

PyObject *type_name = _PyType_FullName(Py_TYPE(obj)); if (name == NULL) { /* do something with this error ... */ PyErr_Format(PyExc_TypeError, "must be str, not %U", type_name); Py_DECREF(name);

When I report an error, I dislike having to handle *new* errors... I prefer that the error handling is done inside PyErr_Format() for me, to reduce the risk of additional bugs.

--

Serhiy also asked if we could expose the same feature at the *Python* level: provide something to get the fully qualified name of a type. It's not just f"{type(obj).__module}.{type(obj).__name__}", but you have to skip the module for builtin types like "str" (not return "builtins.str").

Maybe we can have "name: {0:t}, FQN: {0:T}".format(type(obj)). "t" for name and "T" for fully qualfied name. We would only have to modify type.__format__().

I'm not sure if we need to add new formatters to str % args.

Example of Python code:

raise TypeError("must be str, not %s" % type(fmt).__name__)

I'm not sure about Python changes. My first concern was just to avoid Py_TYPE(obj)->tp_name at the C level. But again, we should keep C and Python consistent. If the behavior of C extensions change, Python modules should be adapted as well, to get the same behavior.

Note: I reverted my change which added the %T formatter from PyUnicode_FromFormatV() to clarify the status of this issue.

I'm not sure about having 2 different, though similar, format codes for 2 similar, though slightly different, cases. (And, for all we know, we might want to use "%t" at some later date for something else.)

Perhaps we could have a single format code plus an optional '#' for the "alternate form":

%T for short form %#T for fully qualified name _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

On 09/11/18 15:23, Victor Stinner wrote:

Hi,

Last week, I opened an issue to propose to add a new %T formatter to PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat() and PyErr_Format():

https://bugs.python.org/issue34595

I merged my change, but then Serhiy Storchaka asked if we can add something to get the "fully qualified name" (FQN) of a type, ex "datetime.timedelta" (FQN) vs "timedelta" (what I call "short" name). I proposed a second pull request to add %t (short) in addition to %T (FQN).

But then Petr Viktorin asked me to open a thread on python-dev to get a wider discussion. So here I am.

After a discussion with Victor. I'll summarize where we are now. There are actually two inconsistencies to fix: - Python modules use `type(obj).__name__` and C extensions use `Py_TYPE(obj)->tp_name`, which inconsistent. - Usage __name__ or __qualname__, and prepending __module__ or not, is inconsistent across types/modules. It turns out that today, when you want to print out a type name, you nearly always want the fully qualified name (including the module unless it's "builtins"). So we can just have "%T" and not "%t". (Or we can add "%t" if a use case arises). It should be possible to do this also in Python, preferably using a name similar to "%T". Most of the usage is in error messages and __repr__, where we don't need to worry about compatibility too much. It should be possible to get the name if you have the type object, but not an instance of it. So, the proposed `PyUnicode_FromFormat("%T", obj)` is incomplete -- if we go that way, we'll also need a function like PyType_GetFullName. Making "%T" work on the type, e.g. `PyUnicode_FromFormat("%T", Py_TYPE(obj))`, would be more general. --- So, I propose adding a "%T" formatter to PyUnicode_FromFormat, to be used like this: PyUnicode_FromFormat("%T", Py_TYPE(obj)) and a "T" format code for type.__format__, to be used like this: f"{type(obj):T}"

On 2018-09-12 02:00, Ethan Furman wrote:

On 09/11/2018 05:21 PM, Barry Warsaw wrote:

...

MRAB wrote on 9/11/18 16:06:

...

...

Perhaps we could have a single format code plus an optional '#' for the "alternate form":

%T for short form %#T for fully qualified name

OTOH, if %T and variants meant "type" but %t mean something entirely different, that *would* probably be confusing.

I think folks used to %-formatting are already used to un-related but similar codes (and related but dissimilar):

- %M for minute - %m for month (or maybe I have that backwards) - %H for 24-hour clock - %I for 12-hour clock - %w for weekday as decimal number - %W for week number of the year

I always have to look it up. :(

Well, for the time of day (24-hour) it's %H:%M:%S, all uppercase.

Hi, For the type name, sometimes, we only get a type (not an instance), and we want to format its FQN. IMHO we need to provide ways to format the FQN of a type for *types* and for *instances*. Here is my proposal: * Add !t conversion to format string * Add ":T" format to type.__format__() * Add "%t" and "%T" formatters to PyUnicode_FromUnicodeV() * Add a read-only type.__fqn__ property # Python: "!t" for instance raise TypeError(f"must be str, not {obj!t}") /* C: "%t" for instance */ PyErr_Format(PyExc_TypeError, "must be str, not %t", obj); /* C: "%T" for type */ PyErr_Format(PyExc_TypeError, "must be str, not %T", mytype); # Python: ":T" for type raise TypeError(f"must be str, not {mytype!T}") Open question: Should we also add "%t" and "%T" formatters to the str % args operator at the Python level? I have a proof-of-concept implementation: https://github.com/python/cpython/pull/9251 Victor Victor

On 9/12/2018 8:33 PM, Victor Stinner wrote:

Hi,

For the type name, sometimes, we only get a type (not an instance), and we want to format its FQN. IMHO we need to provide ways to format the FQN of a type for *types* and for *instances*. Here is my proposal:

* Add !t conversion to format string

I'm strongly opposed to this. This !t conversion would not be widely applicable enough to be generally useful, and would need to be exposed in the f-string and str.format() documentation, even though 99% of programmers would never need or see it. The purpose of the conversions is not to save you from making a function call when you know the type of the arguments. The purpose was specifically to convert arguments to strings so that your format specifier could always use the string formatting mini-language. It was more useful in str.format(), where the format string might be written separately (as user input or a translation, say) and not know the types of the arguments. You can (and I have!) argued that the conversions are completely unneeded in f-strings. raise TypeError(f"must be str, not {obj!t}") Should be written as: raise TypeError(f"must be str, not {type(obj)}")

* Add ":T" format to type.__format__() As you know (I've read the patch) this is just "T". I mention it here for future readers. They should understand that the ":" is a str.format() and f-string construct, and is unknown to __format__().

That said, I think this is a good idea. type.__format__() could also understand "#"  to specify qualname.

* Add "%t" and "%T" formatters to PyUnicode_FromUnicodeV() I think "T" is a good idea, but I think you're adding in obj vs type(obj) just because of the borrowed reference issue in Py_TYPE(). That issue is so much larger than string formatting the type of an object that it shouldn't be addressed here.

* Add a read-only type.__fqn__ property I'm not sure of the purpose of this. When in your examples is it used? # Python: "!t" for instance raise TypeError(f"must be str, not {obj!t}")

/* C: "%t" for instance */ PyErr_Format(PyExc_TypeError, "must be str, not %t", obj);

/* C: "%T" for type */ PyErr_Format(PyExc_TypeError, "must be str, not %T", mytype);

# Python: ":T" for type raise TypeError(f"must be str, not {mytype!T}")

Open question: Should we also add "%t" and "%T" formatters to the str % args operator at the Python level?

No. Again, I think any formatting of type names should not be in a widely used interface, and should remain in our type-specific interface, __format__. %-formatting has no per-type extensibility, and I don't think we should start adding codes for every possible use case. Format codes for datetimes would be way more useful that %t, and I'd be opposed to adding them, too. (I realize my analogy is stretched, because every object has a type. But still.) Eric

I have a proof-of-concept implementation: https://github.com/python/cpython/pull/9251

Victor

Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40tru...

Le jeu. 13 sept. 2018 à 16:01, Eric V. Smith a écrit :

...

* Add !t conversion to format string

I'm strongly opposed to this. This !t conversion would not be widely applicable enough to be generally useful, and would need to be exposed in the f-string and str.format() documentation, even though 99% of programmers would never need or see it.

(I'm thinking aloud.) In the Python code base, I found 115 lines using type(obj).__name__ and 228 lines using obj.__class__.__name__. $ scm.py grep 'type(.*).__name__'|wc -l 115 $ scm.py grep '.__class__.__name__'|wc -l 228 I don't know how to compare these numbers, so I tried to count the number of f-strings: $ git grep '[^"%-]\) x.__class__: --- Moreover, it's also possible to override the "type" symbol in the global or local scope: --- type = id num = 42 print(f"type(num): {type(num)}") # Output: "type(num): 139665950357856" --- One advantage of having a builtin formatter would be to always use internally the builtin type() function to get the type of an object, or not use "type()" in the current scope. The second advantage is to prevent the need of having to decide between type(obj) and obj.__class__ :-)

raise TypeError(f"must be str, not {obj!t}")

Should be written as: raise TypeError(f"must be str, not {type(obj)}")

f"{type(obj)}" behaves as str(type(obj)), but in practice it uses repr(type(obj)):

...

...

f"{type(42)}" ""

My proposed f"{obj!t}" returns the fully qualified name of the object type:

...

...

f"{42!t}" "int"

Do you want to modify str(type) to return a value different than repr(type)? Or maybe it's just a typo and you wanted to write f"{type(obj):T}"?

That said, I think this is a good idea. type.__format__() could also understand "#" to specify qualname.

When I discussed with Petr Viktorin, we failed to find an usecase where __qualname__ was needed. We agreed that we always want the fully qualified name, not just the qualified name.

I think "T" is a good idea, but I think you're adding in obj vs type(obj) just because of the borrowed reference issue in Py_TYPE(). That issue is so much larger than string formatting the type of an object that it shouldn't be addressed here.

Right, that's a side effect of the discussion on the C API. It seems like Py_TYPE() has to go in the new C API. Sorry, the rationale is not written down yet, but Dino convinced me that Py_TYPE() has to go :-)

...

Open question: Should we also add "%t" and "%T" formatters to the str % args operator at the Python level?

No. Again, I think any formatting of type names should not be in a widely used interface, (...)

Ok, that's fine with me :-) Victor

On 9/13/2018 5:52 PM, Petr Viktorin wrote:

On 09/13/18 14:08, Victor Stinner wrote:

...

Le jeu. 13 sept. 2018 à 16:01, Eric V. Smith a écrit :

...

...

* Add !t conversion to format string

I'm strongly opposed to this. This !t conversion would not be widely applicable enough to be generally useful, and would need to be exposed in the f-string and str.format() documentation, even though 99% of programmers would never need or see it.

(I'm thinking aloud.)

In the Python code base, I found 115 lines using type(obj).__name__ and 228 lines using obj.__class__.__name__. [...]

"!t" is not a big improvement over ":T" and "type(obj)".

...

I'm not sure if type(obj) or obj.__class__ should be used, but I can say that they are different: obj.__class__ can be overriden: [...]

Moreover, it's also possible to override the "type" symbol in the global or local scope: [...]

I don't think either of those are problematic. If you override `__class__` or `type`, things will behave weirdly, and that's OK.

...

One advantage of having a builtin formatter would be to always use internally the builtin type() function to get the type of an object, or not use "type()" in the current scope. The second advantage is to prevent the need of having to decide between type(obj) and obj.__class__ :-)

...

raise TypeError(f"must be str, not {obj!t}")

Should be written as: raise TypeError(f"must be str, not {type(obj)}") [...]

Do you want to modify str(type) to return a value different than repr(type)?

Or maybe it's just a typo and you wanted to write f"{type(obj):T}"?

Yes, AFAIK that was a typo.

f'{type(obj)}' becomes type(obj).__format__(''), so you can return something other than __str__ or __repr__ does. It's only by convention that an object's __format__ returns __str__: it need not do so. Eric

...

...

I think "T" is a good idea, but I think you're adding in obj vs type(obj) just because of the borrowed reference issue in Py_TYPE(). That issue is so much larger than string formatting the type of an object that it shouldn't be addressed here.

Right, that's a side effect of the discussion on the C API. It seems like Py_TYPE() has to go in the new C API. Sorry, the rationale is not written down yet, but Dino convinced me that Py_TYPE() has to go :-)

I'll be happy when we get rid of Py_TYPE and get to use moving garbage collectors... but now is not the time. The API for "%T" should be "give me the type". The best way to do that might change in the future.

But at this point, we're bikeshedding. I think all the relevant voices have been heard. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40tru...

On 9/13/2018 8:04 PM, Victor Stinner wrote:

Le ven. 14 sept. 2018 à 00:09, Eric V. Smith a écrit :

...

f'{type(obj)}' becomes type(obj).__format__(''), so you can return something other than __str__ or __repr__ does. It's only by convention that an object's __format__ returns __str__: it need not do so. What's New in Python 3.7 contains:

...

object.__format__(x, '') is now equivalent to str(x) rather than format(str(self), ''). (Contributed by Serhiy Storchaka in bpo-28974.) https://bugs.python.org/issue28974

Oh, I didn't know that a type is free to change this behavior: return something different than str(obj) if the format spec is an empty string.

True! That issue was specific to object.__format__, not any other classes implementation of __format__.

So are you suggesting to change type(obj).__format__('') to return the fully qualified name instead of repr(type)?

I'm not suggesting it, I'm saying it's possible. It indeed might be the most useful behavior.

So "%s" % type(obj) would use repr(), but "{}".format(type(obj)) and f"{type(obj)}" would return the fully qualified name?

"%s" % type(obj) would use str(), not repr. You could either: - keep with convention and have type(obj).__format__('') return type(obj).__str__(), while type(obj).__format__('#') (or what other char you want to use) return the qualname; or - just have type(obj).__format__('') return the qualname, if that's the more useful behavior. Eric