On 21Apr2021 10:14, Paul Moore wrote:

On Wed, 21 Apr 2021 at 09:01, Serhiy Storchaka wrote:

...

Do we need support of more standard conversions? Do we want to support custom conversions (registered globally as encodings and error handlers). re.escape, html.escape and shlex.quote could be very useful in some applications.

That appeals to me just because I like generic features in general, but I'm not sure there are sufficient benefits to justify the complexity for what would basically be a small convenience over calling the function directly.

I frequently use str.format and str.format_map to produce strings from formats provided as configuration; in those methods I can't use arbitrary functions (injection, anyone?) For example: fstags ls -o format-string-here ... to produce a particular listing format. I would _frequently_ like to be able to provide custom conversions. At present I'm using elaborate hacks based on __getattr__ etc to recognise things like this: '{x} is {x_lc} in lowercase' where the _lc suffix is caught and a value computed from "x". Custom conversions would let me use this: '{x} is {x!lc} in lowercase' just by registering 'lc' as a conversion from my code. Chaining then per Serhiy's other suggestion would bring a fair amount of power. Cheers, Cameron Simpson

Cameron Simpson writes:

I would _frequently_ like to be able to provide custom conversions. At present I'm using elaborate hacks based on __getattr__ etc to recognise things like this:

'{x} is {x_lc} in lowercase'

where the _lc suffix is caught and a value computed from "x".

Custom conversions would let me use this:

'{x} is {x!lc} in lowercase'

I don't understand how this is supposed to work. It looks to me like !code is a preprocessor: >>> print(f'{1!a:g}') Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Unknown format code 'g' for object of type 'str' If so, '{x} is {x!lc:foo} in lowercase' will fail because str doesn't implement the 'foo' format code. Do we really need to extend format() rather than using def lc(x): return str(x).lower() '{x} is {lc(x)} in lowercase' ?

On 23Apr2021 18:25, Stephen J. Turnbull wrote:

Cameron Simpson writes:

...

I would _frequently_ like to be able to provide custom conversions. At present I'm using elaborate hacks based on __getattr__ etc to recognise things like this:

'{x} is {x_lc} in lowercase'

where the _lc suffix is caught and a value computed from "x".

Custom conversions would let me use this:

'{x} is {x!lc} in lowercase'

I don't understand how this is supposed to work. It looks to me like !code is a preprocessor: [...] If so,

'{x} is {x!lc:foo} in lowercase'

will fail because str doesn't implement the 'foo' format code.

Maybe we're talking about different things. In the example above, I'm talking about "lc", not "foo".

Do we really need to extend format() rather than using

def lc(x): return str(x).lower()

'{x} is {lc(x)} in lowercase'

I'm not talking about f'' strings, but str.format_map. They use the same {} placeholder syntax, but for obvious injection related reasons you can't call an arbitrary function from str.format_map. (You can play games with magic __getattr__ based attributes, like: {z.some_attr} My use case is presupplied strings, eg a command line supplied format string. So _not_ an f'blah {lc(x)}' string, but a plain string using format_map. I'm enjoying using Python style string formats, and who wants to write yet another macro syntax? Contrast: Python 3.9.2 (default, Feb 24 2021, 13:30:36) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> def lc(x): return x.lower() ... >>> z='ABC' >>> f'{lc(z)}' 'abc' >>> fmt='{lc(z)}' >>> fmt.format_map({z:"ABC"}) Traceback (most recent call last): File "<stdin>", line 1, in <module> KeyError: 'lc(z)' If I could provide additional formatters to format_map to extend beyond 'r' for repr and friends, I could go: fmt.format_map({z:"ABC"}, conversion_map={'lc':lambda str.lower}) and use: fmt = '{z!lc}' Cheers, Cameron Simpson

On Sat, Apr 24, 2021 at 11:36 PM Stephen J. Turnbull wrote:

Assuming that *exact* use case, wouldn't

>>> class LowerableStr(str): ... def __format__(self, fmt): ... if fmt == 'lc': ... return self.lower() ... else: ... return str.__format__(self, fmt) ... >>> "{x} is {x:lc} in lowercase".format_map({'x' : LowerableStr("This")}) 'This is this in lowercase'

do? For generic x you'd have to do something a bit more complicated, but the basic strategy would be the same. Similarly, it shouldn't be hard to design a fairly generic wrapper class that allows you to map format codes to arbitrary functions.

Now also add in that you will need uppercase and titlecase, and then allow actual formatting of the string as well as case selection. Then make sure that every message you format can have every piece of text supporting this, since you'll never know exactly what parts of what messages may need this sort of manipulation. It won't look nearly as simple any more. ChrisA

On 24Apr2021 22:35, Stephen J. Turnbull wrote:

Cameron Simpson writes:

...

On 23Apr2021 18:25, Stephen J. Turnbull wrote:

...

I don't understand how this is supposed to work. It looks to me like !code is a preprocessor: [...] If so,

'{x} is {x!lc:foo} in lowercase'

will fail because str doesn't implement the 'foo' format code.

Maybe we're talking about different things. In the example above, I'm talking about "lc", not "foo".

I don't think so. I know you want to talk only about "lc", but I want to understand how this interacts with "foo", and why you can't use "foo" in your application. [...]

...

My use case is presupplied strings, eg a command line supplied format string.

In that case the format string is user input, and x is a variable in the program that the user can have substituted into their string?

Assuming that *exact* use case, wouldn't

...

...

...

class LowerableStr(str): ... def __format__(self, fmt): ... if fmt == 'lc': ... return self.lower() ... else: ... return str.__format__(self, fmt) ... "{x} is {x:lc} in lowercase".format_map({'x' : LowerableStr("This")}) 'This is this in lowercase'

do?

You're perfectly correct. ":lc" can be shoehorned into doing what I ask. But __format__ is in the wrong place for how I'd like to do this. First up, I have somehow missed this (":format_name") in the semirecursive mess which is the python format-and-friends descriptions. (object.__format__? str.format? str.formap_map? f''? the format mini-language? all in separate places, for reasonable reasons, but my head has exploded multiple times trying to stitch them together). On reflection, __format__ is something I may already have considered and rejected. Let me explain. The "!r" conversion is applied _by_ the formatting, _to_ an arbitrary value. So I could write a single function for my hypothetical "!lc" which did what I want for an arbitrary object, because it would be called with the value being converted. By contrast, the ":lc" format specifier form requires the value being formatted _itself_ to have a special __format__ method. This scales poorly when I might put almost anything into the format string. I'm not speaking here of allowing an end user to inject arbitrary access code into my programme via a format string, but that the format strings I'm using via .format_map(mapping) are given a mapping which is a pretty rich view of an almost arbitrary data structure I've made available for display via the format string. In case you care, my primary use case is a tag library _with_ an ontology, where tag values can be arbitrary Python values. (In reality, those values need to be JSON renderable just now, since they land in text files or database JSON blobs when persisted.) Anyway, the format string lets me write formats like: track {track_id} has artist {artist._meta.fullname} which hops off through the ontology, or: # from a config or command line or default # I like lowercased filenames a lot filename_format = '{artist_lc}--{album_lc}--{track_id}--{title_lc}.mp3' with open(filename_format.format_map(tagset.ns()), 'wb'): ... write file data ... The filename_format is the example where I want some kind of "lc" conversion/formatting to apply to an arbitrary value. In the code above, tagset.ns() returns a magic subclass of SimpleNamespace which has the following properties: - it allows mapping-like attribute access so that I can pass it to format_map() - it has attributes/keys computed from the tags in tagset so that I can use them in the format string - it has an elaborate __getattr__ method recognising a number of suffixes like "_lc" - should there be no actual tag of that name it will find the prefix and lowercase that, for example - a tag title="My Name" gets "My Name" from {title} and "my_name" from {title_lc} - the same __getattr__ recognises some things like _meta and returns another namespace containing metadata from the ontology, letting me say: {artist._meta.fullname} All this is to support giving the user/config a fairly rich suite of stuff which can go in a format string using the Python format string syntax. Regarding "!lc" vs ":lc": The "!lc" approach: If the format syntax let one supply a mapping of conversions to functions for something like "!lc" then I could rip out a big chunk of complexity from __getattr__, because the "_lc" suffixes above are essentially a syntax hack to work around that shortcoming. The ":lc" approach: The problem with __format__ is that it must be applied to a class. The values inside {foo.bar.zot} in the format string might be almost any type. The only way to get __format__ to do what I'd like is to wrap every such value in a proxy of some kind with a .__format__ method. In principle I can do that in my magic namespace class (from .ns() above). But that's yet another layer of complexity in something I'm already unhappy with. Hence the word "shoehorn" earlier. So to my mind, being able to plug in a mapping of additional (or overriding) conversion specifiers would be the most natural way to improve my situation. Architecturally, that is where I would want my magic "lc" et al to land. I could move the magic "._meta" attribute in there as well producing fewer "magic" attributes, etc. That is why I'm for being able to augument the "!conversion" stuff. Cheers, Cameron Simpson

On 25Apr2021 10:54, Cameron Simpson wrote:

On 24Apr2021 22:35, Stephen J. Turnbull wrote: [...]

...

...

My use case is presupplied strings, eg a command line supplied format string.

In that case the format string is user input, and x is a variable in the program that the user can have substituted into their string?

Assuming that *exact* use case, wouldn't

...

...

...

class LowerableStr(str): ... def __format__(self, fmt): ... if fmt == 'lc': ... return self.lower() ... else: ... return str.__format__(self, fmt) ... "{x} is {x:lc} in lowercase".format_map({'x' : LowerableStr("This")}) 'This is this in lowercase'

do?

You're perfectly correct. ":lc" can be shoehorned into doing what I ask. But __format__ is in the wrong place for how I'd like to do this.

Just to follow up on this, I've been experimenting with Stephen's suggestion of using ":foo" style format specifiers. With enough success that I'm probably going to run with it is I can make the implementation clean enough. I'm currently making a subclass of the Formatter builtin class, which supplies the string parser and lets one override various methods to implemenent the "{...}" parts. Currently I can do this: I've got a little ontology, saying that a tag named "colour" is "a colour, a hue" and expects to be a Python str. It also has metadata for the particular colour "blue". { 'type.colour': TagSet:{'description': 'a colour, a hue', 'type': 'str'}, 'meta.colour.blue': TagSet:{'url': 'https://en.wikipedia.org/wiki/Blue', 'wavelengths': '450nm-495nm'} } And I've got a tag set describing some object which is blue: {'colour': 'blue', 'labels': ['a', 'b', 'c'], 'size': 9} The TagSet class has a .format_as(format_string) method which hooks into the Formatter subclass and formats a string according to the TagSet. A little test programme to write information into some command line supplied format strings: [~/hg/css-tagsets(hg:tagsets)]fleet2*> py3 -m cs.tagset '{colour}' '{colour:meta}' '{colour:meta.url}' {colour} => 'blue' {colour:meta} => 'url="https://en.wikipedia.org/wiki/Blue" wavelengths="450nm-495nm"' {colour:meta.url} => 'https://en.wikipedia.org/wiki/Blue' Looking promising to me. Cheers, Cameron Simpson

On 06May2021 03:43, Dennis Sweeney wrote:

Maybe I'm missing something, but why do you need the SimpleNamespace at all? Why not make your own mapping as in

class StringMapper: ... def __getitem__(self, s): # Whatever arbitrary behavior you want # Process suffixes, etc here, for example: if s.endswith(".lc"): return self.wrapped[s.removesuffix(".lc")].lower() return self.wrapped[s]

format_string.format_map(StringMapper(whatever))

Maybe then the data can just be data and this wrapper can handle all the formatting conversions.

All this is somewhat orthoganal to adding more !foo things. If we're not talking about extending the conversions, we should probably take this over to python-list, because the ":foo" stuff is just implementation. That said, because my tags can have dots in their names and I want to be able to write: {tag.name.with.dot} and this is accomplished with an elaborate SimpleNamespace subclass to make the .name.with.dot stuff work as attributes. Just don't ask :-( Oh, you did :-) Anyway, I may be shifting sideways per my recent ":foo" post, using a new class which is a magic view of the TagSet and a Formatter subclass which parses the field_names itself (thus grabbing the dotted identifier). Still a work in progress. Cheers, Cameron Simpson

On 4/23/2021 5:22 AM, Stephen J. Turnbull wrote:

But we could change int.__format__ to allow 's' as a format code[1], automagically calling str(), just as 'efg' are allowed and automagically call float(). ... [1] I recognize this may imply many changes, since some (all?) of the builtin types seem to be supported by object.__format__.

object.__format__ doesn't do anything except call str() (assuming the format spec is not specified). It's each builtin type that supports formatting itself. There's an int.__format__, str.__format__, float.__format__, datetime.datetime.__format__, decimal.__format__, etc. Many of these use some shared code to do the heavy lifting, and many of them share the same format spec, or much of it. But conceptually they're all distinct and each one can interpret the format spec however they choose. For example, int.__format__ chose not to support the "s" type. And datetime.datetime.__format__ chose to call .strftime(). It's unfortunate that PEP 3101 says "If an object does not define its own format specifiers, a standard set of format specifiers is used", because it's not true. It should say that "Many types will chose to support a standard format specifier". Eric

Serhiy Storchaka writes:

Because it converts value to string, and string formatting does not support "g". Converters !s, !r and !a are separated from format specifier, and it is old and widely used feature.

And poorly documented, IMO. I'll see if I can do better.

but if we are going to make this feature public, it is worth to discuss it ahead to avoid name conflicts in future. I asked what letters should be chosen for convertors for int() and index().

I would go with !f for "float", !d for "decimal" (int) and !x or !i for "index". I like !x better because of the ambiguity of "i". But I don't think I'm likely to use any of them, so that's about +0.2 for all of them.

I am not sure which relation does it have to I18N.

First, the use of "_()" as the localizable string marker conflicts with the common use of "_" as a "throwaway" identifier. Second, POSIX locale lookup (and possibly gettext, too) seems like a very obvious candidate for !conversion before formatting a string field, similar to casting a number to float for e, f, or g formatting.

We need to format a value as integer or float independently from __format__ implementation, and raise an error if it cannot be converted to integer or float. The purpose of the feature is bypassing __format__ and get the same result as in printf-style formatting.

Surely you don't mean "bypass format"? If you bypass format, how do you select among e, f, and g? This is exactly the confusion that caused me trouble: the use cases for !s, !r, and !a generally do bypass __format__ (but something like "x!r:>10" is occasionally useful and does work). So at the language level they appear to be syntactic sugar, since they are not programmable (see Cameron Simpson's posts).

Yes, we could add support for "s" in int.__format__, but it was decided to not do this for some reasons.

I'm not really suggesting that we do this *now*, since the backward incompatibility is horrifying. I'm asking why converters were exposed in the first place since they look like syntactic suger. I guess the basic answer is "they're already needed for int -> float conversion, and they're nice abbreviations for str(), repr(), and ascii() because format() and friends don't allow function calls in {}". Anyway, I'm satisfied with that, you don't need to convince me.

Supporting the "h" format code in the __format__ method for URL objects is a reasonable idea, and it is the purpose of __format__ methods. But it does not relates to converters. If you want to dump the value of some variable using repr(), do you want to add support of "r" in every implementation of __format__ in the world (and what if some is not support it or use it with different semantic)? "%r" % x just calls repr(), and we wanted this feature in new formatting.

You could have it if the format codes 's', 'r', and 'a' were implemented in the formatting mechanisms, i.e., in print() itself, in format(), in str.format, and in f-strings. I'm not ready to argue for changing it now, but I think it would have been better if format specs included the preprocessor !conversion and also a postprocessor (for Cameron), and maybe allowed those to be programmable via a mapping of codes to type converters for the preprocessor and a mapping of codes to str -> str functions for the postprocessor, or passed these to __format__ as a triple (format_spec, pre_code, post_code). Steve