Combine f-strings with i18n
I have recently updated my code to use the more pythonic f-string instead of '{}'.format() Now, I want to start on the road to multilingual internationalization, and I run into two problems. The first problem is that f-strings do not combine with i18n. I have to revert to the '{}'.format() style. The second problem is that I need to translate strings on the fly. I propose to add a f''.language() method to the f-string format. Example: user = 'Pedro' f'Hi {user}' would be translated to 'Hola Pedro' if the locale were set to Spanish. f'Hi {user}'.language('es_ES') would be translated in the same way. To extract translatable strings from the source code, the source code could contain a 'HAS_LOCALES' flag (or something similar) at the top of the code. This way, the pygettext.py program would know that translatable f-strings are within the code. Rationale: More pythonic. At this moment, _('').format() is the way to go, so I would need to wrap another call around that: T(_(''), args, 'es_ES') <===This is an ugly hack. # Set the _() function to return the same string _ = lambda s: s es = gettext.translation('myapplication', languages=['es_ES']) def T(translatable_string, args_dictionary = None, language = None) if 'es_ES' == language: # Return translated, formatted string return es.gettext(translatable_string).format(args) # Default, return formatted string return translatable_string.format(args)
On Fri, Sep 14, 2018 at 7:02 PM, Hans Polak <hpolak@polak.es> wrote:
I have recently updated my code to use the more pythonic f-string instead of '{}'.format()
Well there's your problem right there. Don't change your string formatting choice on that basis. F-strings aren't "more Pythonic" than either .format() or percent-formatting; all three of them are supported for good reasons. For i18n, I think .format() is probably your best bet. Trying to mess with f-strings to give them methods is a path of great hairiness, as they are not actually objects (they're expressions). ChrisA
On 17/09/18 09:53, Niki Spahiev wrote:
Is it possible to use f-strings when making multilingual software? When i write non-hobby software translation is hard requirement.
At this moment, it seems that this is not possible. My use case is not very unique and that's why I wrote the proposal in the first place. I'm working on a web server / application server. On the web, you have to take the users preferences into account, including language. If a user has the navigator configured for English, I have to return English (if I am doing i18n). That's why I would like to see a parameter that can be passed to the f-string. I don't think this should be too problematic, really. pygettext.py extracts strings surrounded by _('') My proposal would be to do that with f-strings. Let pygettext.py extract f-strings. The compiler can then rewrite these to normal unicode strings. For instance: f'Hi {user}'.language('es') would become T(_('Hi {user}'), 'es', user=user) My first email had pseudo-code. This is my working function. def T(translatable_string, language=None, *args, **kwargs): if args: print(args) if 'es' == language: # Return translated, formatted string return es.gettext(translatable_string).format(**kwargs) # Default, return formatted string return translatable_string.format(**kwargs)
Hans Polak writes:
No, it's not possible.
If a user has the navigator configured for English, I have to return English (if I am doing i18n).
This is understood. Nobody is telling you what you want is an unreasonable desire. I'm telling you that I'm pretty sure your proposed syntax isn't going to happen, because it requires deep changes in the way Python evaluates expressions as far as I can see.
That's why I would like to see a parameter that can be passed to the f-string.
This doesn't make sense to me. Such configurations are long-lasting. In this context, the POSIX model (where the target language, or priority list of languages, is configured in the environment) is reasonable. There's no good reason for passing the language every time a string is formatted throughout an interaction with such a user. What we want is a way to tell the f-string to translate itself, and optionally specify a language.
I don't think this should be too problematic, really.
Your proposal to use method syntax is *definitely* problematic. The f-string is an expression, and must be evaluated to a str first according to the language definition.
It could, but it's not going to. Implementing that with a reasonable amount of backward compatibility requires two tokens of lookahead and a new keyword as far as I can see. The problem is that unless the .language method is invoked on the f-string in the same expression, the f-string needs to be converted to a string immediately, as it is in Python 3.6 and 3.7. To decide whether to do this in the case where there is a method invoked, the parser needs to read the f-string (lookahead = 0), the "." (lookahead = 1), and the token "language" (lookahead = 2), and then for the parser to know that this is the special construct, it needs to know the special token. "Keyword" in this context simply means "a token that the parser knows about", but in general in Python we want keywords to be reserved to the language, which is considered a very high cost. What could work is an extension to the formatting language. I suggest abusing the *conversion flag*. (It's an abuse because I'm going to apply it to the whole f-string, while the current Language Reference says it's applied to the value being formatted.[1]) This flag would only be allowed as the first item in the string. The idea is that `f"{lang!g}Hello, {user}!"` would be interpreted as _ = get_gettext(lang) _("Hello, {user}!").format(user=user) and `f"{!g}Hello, {user}!"` as `_("Hello, {user}!").format(user=user)`, reusing the the most recent value of `_`. The "g" in "!g" stands for "gettext", of course. GNU xgettext can be taught to recognize things like `f"{...!g}` as translatable string markers; I'm sure pygettext can too. I'm assuming the implementation of get_gettext from my earlier post, reproduced at the end for reader convenience. One warning about this syntax: I think the gettext module is a pretty popular way to implement message localization. However, I'm not sure it's the only way, and I would guess that folks who use something else would want to be able to use f-strings with that package, too. So there may need to be a way to configure the translation engine.
Footnotes: [1] I'm not sure whether "the g conversion is applied to the 'lang' variable, and the effect is to set the global 'translate' function" is pure sophistry or not, but I don't find it convincing. YMMV
Eric V. Smith writes:
See also PEP 501, which could be used for i18n.
I don't see how this immediately helps the OP, who wants a *literal* expression that automatically invokes the translation machinery as well as the interpolation machinery. The translation machinery needs access to the raw template string, which is what (human) translators will be provided by pygettext. But as far as I can see there is nothing in i-string processing that provides a hook for this. It presumably wouldn't be impossible to provide a class or factory function derived from InterpolationTemplate that transparently does the lookup, and then constructs an InterpolationTemplate (duplicating the compiler's work?) But a direct compilation to InterpolationTemplate is hard-wired for (literal) i-strings. AFAICS this is the *only* benefit of PEP 501 over simply defining the InterpolationTemplate class in a library module, and it's not usable by gettext-style I18N! We could add a `translate` method, to completely replace the raw_template *before* the compiler parses it into parsed_template. But this gets complicated in the sense of the Zen, because we want to be able to change the translate method on the fly (if the target language changes, or to change the gettext domain, or whatever), or if we have non-I18N uses for i-strings in our application. I hope I'm missing something! Steve
Another way forward could be a preprocessor. All this can be done with a fairly simple script using parso. If the op is interested I could whip out a prototype. The cool thing about parso is that it’s a round trip AST so it’s easy to perform refactorings or in this case preprocessing without affecting formatting or comments. / Anders
translated at compile time. It should be translated at run-time. There are three situations. 1. No translation, just a regular f-string. 2. App translation. The f-string gets translated to the configured language. 3. On the fly translation. The string gets translated to the language passed as an argument as required. In code, this would be. 1. f'Hi {user}' 2. f'{!g}Hi {user}' 3. f'{lang!g}Hi {user}' Cases 2 and 3 need some additional code, just like with gettext. I'm sorry if that wasn't clear from the start. All I want is the code to be simpler to write and maintain. I do not want to have complicated parsing for the compiler.
Another way forward could be a preprocessor. All this can be done with a fairly simple script using parso. This is probably the idea.
Cheers, Hans
On 18/09/2018 08:59, Hans Polak wrote:
Surely the simpler solution is to specify in I18n any items within un-escaped {} pairs is excluded from the translation, lookups, etc., and that translation needs to take place, also leaving the {} content alone, before f string processing. Other than that there is no change. So: _(f'Hi {user}') would be in the .po/.mo as just 'Hi ' and if our locale is set to FR this gets translated to f'Bonjor {user}' which then gets the user variable substituted in. If you wanted to insert into an f string a value that is itself subject to I18n you need to mark the content assigned to that value for translation. For example: parts_of_day = [_("Morning"), _("Afternoon"), _("Evening"), _("Night"), ] tod = lookup_time_as_pod() greeting = _(f"Good {tod}") If our locale happens to be a German one and our current time of day is morning then tod will be assigned as "morgan" and our greeting will be "Gutten Morgan", etc. This should work without any major problems whether our locale is fixed at start-up or changes dynamically. As far as I can see the only possibly required change to the core python language is that the evaluation order may need to be able to be override-able so that the translate function, (with the leave {.*} alone rule), is called _before_ the f string formatting, (I think that with current precedence it would not). Is there, or could there be, an "@eager" or "@push_precedence" decorator, or some such, that could be added to translate so as to do this? The remaining changes would be in the translate/I18n package(s) and the documents of course. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. https://www.avg.com
On Wed, Sep 19, 2018 at 3:55 PM Steve Barnes <gadgetsteve@live.co.uk> wrote:
How about this: Have a script that runs over your code, looking for "translatable f-strings": _(f'Hi {user}') and replaces them with actually-translatable strings: _('Hi %s') % (user,) _('Hi {user}').format(user=user) Take your pick of which way you want to spell it. Either of these is easily able to be picked up by a standard translation package, is 100% legal Python code in today's interpreters, and doesn't require any bizarre markers and such saying that things need to be processed out of order (the parentheses specify the order for you). Not everything has to be an f-string. ChrisA
On Wed, Sep 19, 2018 at 4:52 PM Anders Hovmöller <boxed@killingar.net> wrote:
Ah. It wasn't clear what your destination was, so I thought you were talking about doing the translation itself using parso. But yeah, grab one of these sorts of parsing libraries, do the transformation, save back, then use a standard translation library. Seems a lot easier than changing the language. ChrisA
Ah, my bad. I agree that this is the way forward for people who are trying to localize an existing app, but I still think we should _also_ change the language. F-strings are great and .format is powerful but there is a too big gap in usability and readability between them. This gap is one of the most compelling motivations for my suggestion of a short form for keyword arguments, while also helping us poor guys who deal with huge legacy code bases :) / Anders
On 17/09/18 19:42, Stephen J. Turnbull wrote:
# At the start of the app, the languages get loaded in memory. translate = translation('app','.locale') translate.install() es = translation('app','.locale',languages=['es']) es.install() # Get the preferred user language from the http request T(_('Hello {user}...'), user_language,user=user) def T(translatable_string, language=None, *args, **kwargs): if 'es' == language: # Return translated, formatted string return es.gettext(translatable_string).format(**kwargs) # Default, return formatted string return translatable_string.format(**kwargs)
Such configurations are long-lasting. If it is for the whole app, yes. Not if it is just the request.
1. No translation, just a regular f-string. 2. App translation. The f-string gets translated to the configured language. Long lasting configuration. 3. On the fly translation. The string gets translated to the language passed as an argument as required.
Cheers, Hans
On Mon, Sep 17, 2018 at 5:53 PM, Niki Spahiev <niki.spahiev@gmail.com> wrote:
I won't say it's *impossible*, but it's certainly not what I would recommend. Use one of the other formatting methods (percent formatting or the .format() method), since they start with a single string object rather than an expression. Don't assume that f-strings should be used for everything just because they're newer. They have their place, and that place is NOT translation. ChrisA
Hans Polak writes:
The second problem is that I need to translate strings on the fly.
I don't understand what that means. gettext() does exactly that. Do you mean you need to switch languages on the fly?
I don't think so, since as Chris points out, an f-string is an expression whose value is a str, with the values of the locals already interpolated in the string. You'd need a lot of magic in the compiler to make this work.
If, as I guessed, you want to change languages on the fly, I would advise to make a hash table of languages used so far, and add translation tables to it on the fly as new languages are requested. Or, if space is at a premium, a LRU cache of tables.
Then you can replace this with # Use duck-typing of gettext.translation objects class NullTranslation: def __init__(self): self.gettext = lambda s: s def get_gettext(language, translation={'C': NullTranslation()}): if language not in translation: translation[language] = \ gettext.translation('myapplication', languages=[language]) return translation[language].gettext and # This could be one line, but I guess in many cases you're likely # to use use the gettext function repeatedly. Also, use of the # _() idiom marks translatable string for translators. _ = get_gettext(language) _(translatable_string).format(key=value...) instead of T(translatable_string, args, language) which isn't great style (the names "T" and "args" are not very evocative). It's a little more code, but it doesn't require changing the semantics of existing Python code, and is prettier than your T() function IMO. Steve
On Fri, Sep 14, 2018 at 7:02 PM, Hans Polak <hpolak@polak.es> wrote:
I have recently updated my code to use the more pythonic f-string instead of '{}'.format()
Well there's your problem right there. Don't change your string formatting choice on that basis. F-strings aren't "more Pythonic" than either .format() or percent-formatting; all three of them are supported for good reasons. For i18n, I think .format() is probably your best bet. Trying to mess with f-strings to give them methods is a path of great hairiness, as they are not actually objects (they're expressions). ChrisA
On 17/09/18 09:53, Niki Spahiev wrote:
Is it possible to use f-strings when making multilingual software? When i write non-hobby software translation is hard requirement.
At this moment, it seems that this is not possible. My use case is not very unique and that's why I wrote the proposal in the first place. I'm working on a web server / application server. On the web, you have to take the users preferences into account, including language. If a user has the navigator configured for English, I have to return English (if I am doing i18n). That's why I would like to see a parameter that can be passed to the f-string. I don't think this should be too problematic, really. pygettext.py extracts strings surrounded by _('') My proposal would be to do that with f-strings. Let pygettext.py extract f-strings. The compiler can then rewrite these to normal unicode strings. For instance: f'Hi {user}'.language('es') would become T(_('Hi {user}'), 'es', user=user) My first email had pseudo-code. This is my working function. def T(translatable_string, language=None, *args, **kwargs): if args: print(args) if 'es' == language: # Return translated, formatted string return es.gettext(translatable_string).format(**kwargs) # Default, return formatted string return translatable_string.format(**kwargs)
Hans Polak writes:
No, it's not possible.
If a user has the navigator configured for English, I have to return English (if I am doing i18n).
This is understood. Nobody is telling you what you want is an unreasonable desire. I'm telling you that I'm pretty sure your proposed syntax isn't going to happen, because it requires deep changes in the way Python evaluates expressions as far as I can see.
That's why I would like to see a parameter that can be passed to the f-string.
This doesn't make sense to me. Such configurations are long-lasting. In this context, the POSIX model (where the target language, or priority list of languages, is configured in the environment) is reasonable. There's no good reason for passing the language every time a string is formatted throughout an interaction with such a user. What we want is a way to tell the f-string to translate itself, and optionally specify a language.
I don't think this should be too problematic, really.
Your proposal to use method syntax is *definitely* problematic. The f-string is an expression, and must be evaluated to a str first according to the language definition.
It could, but it's not going to. Implementing that with a reasonable amount of backward compatibility requires two tokens of lookahead and a new keyword as far as I can see. The problem is that unless the .language method is invoked on the f-string in the same expression, the f-string needs to be converted to a string immediately, as it is in Python 3.6 and 3.7. To decide whether to do this in the case where there is a method invoked, the parser needs to read the f-string (lookahead = 0), the "." (lookahead = 1), and the token "language" (lookahead = 2), and then for the parser to know that this is the special construct, it needs to know the special token. "Keyword" in this context simply means "a token that the parser knows about", but in general in Python we want keywords to be reserved to the language, which is considered a very high cost. What could work is an extension to the formatting language. I suggest abusing the *conversion flag*. (It's an abuse because I'm going to apply it to the whole f-string, while the current Language Reference says it's applied to the value being formatted.[1]) This flag would only be allowed as the first item in the string. The idea is that `f"{lang!g}Hello, {user}!"` would be interpreted as _ = get_gettext(lang) _("Hello, {user}!").format(user=user) and `f"{!g}Hello, {user}!"` as `_("Hello, {user}!").format(user=user)`, reusing the the most recent value of `_`. The "g" in "!g" stands for "gettext", of course. GNU xgettext can be taught to recognize things like `f"{...!g}` as translatable string markers; I'm sure pygettext can too. I'm assuming the implementation of get_gettext from my earlier post, reproduced at the end for reader convenience. One warning about this syntax: I think the gettext module is a pretty popular way to implement message localization. However, I'm not sure it's the only way, and I would guess that folks who use something else would want to be able to use f-strings with that package, too. So there may need to be a way to configure the translation engine.
Footnotes: [1] I'm not sure whether "the g conversion is applied to the 'lang' variable, and the effect is to set the global 'translate' function" is pure sophistry or not, but I don't find it convincing. YMMV
Eric V. Smith writes:
See also PEP 501, which could be used for i18n.
I don't see how this immediately helps the OP, who wants a *literal* expression that automatically invokes the translation machinery as well as the interpolation machinery. The translation machinery needs access to the raw template string, which is what (human) translators will be provided by pygettext. But as far as I can see there is nothing in i-string processing that provides a hook for this. It presumably wouldn't be impossible to provide a class or factory function derived from InterpolationTemplate that transparently does the lookup, and then constructs an InterpolationTemplate (duplicating the compiler's work?) But a direct compilation to InterpolationTemplate is hard-wired for (literal) i-strings. AFAICS this is the *only* benefit of PEP 501 over simply defining the InterpolationTemplate class in a library module, and it's not usable by gettext-style I18N! We could add a `translate` method, to completely replace the raw_template *before* the compiler parses it into parsed_template. But this gets complicated in the sense of the Zen, because we want to be able to change the translate method on the fly (if the target language changes, or to change the gettext domain, or whatever), or if we have non-I18N uses for i-strings in our application. I hope I'm missing something! Steve
Another way forward could be a preprocessor. All this can be done with a fairly simple script using parso. If the op is interested I could whip out a prototype. The cool thing about parso is that it’s a round trip AST so it’s easy to perform refactorings or in this case preprocessing without affecting formatting or comments. / Anders
translated at compile time. It should be translated at run-time. There are three situations. 1. No translation, just a regular f-string. 2. App translation. The f-string gets translated to the configured language. 3. On the fly translation. The string gets translated to the language passed as an argument as required. In code, this would be. 1. f'Hi {user}' 2. f'{!g}Hi {user}' 3. f'{lang!g}Hi {user}' Cases 2 and 3 need some additional code, just like with gettext. I'm sorry if that wasn't clear from the start. All I want is the code to be simpler to write and maintain. I do not want to have complicated parsing for the compiler.
Another way forward could be a preprocessor. All this can be done with a fairly simple script using parso. This is probably the idea.
Cheers, Hans
On 18/09/2018 08:59, Hans Polak wrote:
Surely the simpler solution is to specify in I18n any items within un-escaped {} pairs is excluded from the translation, lookups, etc., and that translation needs to take place, also leaving the {} content alone, before f string processing. Other than that there is no change. So: _(f'Hi {user}') would be in the .po/.mo as just 'Hi ' and if our locale is set to FR this gets translated to f'Bonjor {user}' which then gets the user variable substituted in. If you wanted to insert into an f string a value that is itself subject to I18n you need to mark the content assigned to that value for translation. For example: parts_of_day = [_("Morning"), _("Afternoon"), _("Evening"), _("Night"), ] tod = lookup_time_as_pod() greeting = _(f"Good {tod}") If our locale happens to be a German one and our current time of day is morning then tod will be assigned as "morgan" and our greeting will be "Gutten Morgan", etc. This should work without any major problems whether our locale is fixed at start-up or changes dynamically. As far as I can see the only possibly required change to the core python language is that the evaluation order may need to be able to be override-able so that the translate function, (with the leave {.*} alone rule), is called _before_ the f string formatting, (I think that with current precedence it would not). Is there, or could there be, an "@eager" or "@push_precedence" decorator, or some such, that could be added to translate so as to do this? The remaining changes would be in the translate/I18n package(s) and the documents of course. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. https://www.avg.com
On Wed, Sep 19, 2018 at 3:55 PM Steve Barnes <gadgetsteve@live.co.uk> wrote:
How about this: Have a script that runs over your code, looking for "translatable f-strings": _(f'Hi {user}') and replaces them with actually-translatable strings: _('Hi %s') % (user,) _('Hi {user}').format(user=user) Take your pick of which way you want to spell it. Either of these is easily able to be picked up by a standard translation package, is 100% legal Python code in today's interpreters, and doesn't require any bizarre markers and such saying that things need to be processed out of order (the parentheses specify the order for you). Not everything has to be an f-string. ChrisA
On Wed, Sep 19, 2018 at 4:52 PM Anders Hovmöller <boxed@killingar.net> wrote:
Ah. It wasn't clear what your destination was, so I thought you were talking about doing the translation itself using parso. But yeah, grab one of these sorts of parsing libraries, do the transformation, save back, then use a standard translation library. Seems a lot easier than changing the language. ChrisA
Ah, my bad. I agree that this is the way forward for people who are trying to localize an existing app, but I still think we should _also_ change the language. F-strings are great and .format is powerful but there is a too big gap in usability and readability between them. This gap is one of the most compelling motivations for my suggestion of a short form for keyword arguments, while also helping us poor guys who deal with huge legacy code bases :) / Anders
On 17/09/18 19:42, Stephen J. Turnbull wrote:
# At the start of the app, the languages get loaded in memory. translate = translation('app','.locale') translate.install() es = translation('app','.locale',languages=['es']) es.install() # Get the preferred user language from the http request T(_('Hello {user}...'), user_language,user=user) def T(translatable_string, language=None, *args, **kwargs): if 'es' == language: # Return translated, formatted string return es.gettext(translatable_string).format(**kwargs) # Default, return formatted string return translatable_string.format(**kwargs)
Such configurations are long-lasting. If it is for the whole app, yes. Not if it is just the request.
1. No translation, just a regular f-string. 2. App translation. The f-string gets translated to the configured language. Long lasting configuration. 3. On the fly translation. The string gets translated to the language passed as an argument as required.
Cheers, Hans
On Mon, Sep 17, 2018 at 5:53 PM, Niki Spahiev <niki.spahiev@gmail.com> wrote:
I won't say it's *impossible*, but it's certainly not what I would recommend. Use one of the other formatting methods (percent formatting or the .format() method), since they start with a single string object rather than an expression. Don't assume that f-strings should be used for everything just because they're newer. They have their place, and that place is NOT translation. ChrisA
Hans Polak writes:
The second problem is that I need to translate strings on the fly.
I don't understand what that means. gettext() does exactly that. Do you mean you need to switch languages on the fly?
I don't think so, since as Chris points out, an f-string is an expression whose value is a str, with the values of the locals already interpolated in the string. You'd need a lot of magic in the compiler to make this work.
If, as I guessed, you want to change languages on the fly, I would advise to make a hash table of languages used so far, and add translation tables to it on the fly as new languages are requested. Or, if space is at a premium, a LRU cache of tables.
Then you can replace this with # Use duck-typing of gettext.translation objects class NullTranslation: def __init__(self): self.gettext = lambda s: s def get_gettext(language, translation={'C': NullTranslation()}): if language not in translation: translation[language] = \ gettext.translation('myapplication', languages=[language]) return translation[language].gettext and # This could be one line, but I guess in many cases you're likely # to use use the gettext function repeatedly. Also, use of the # _() idiom marks translatable string for translators. _ = get_gettext(language) _(translatable_string).format(key=value...) instead of T(translatable_string, args, language) which isn't great style (the names "T" and "args" are not very evocative). It's a little more code, but it doesn't require changing the semantics of existing Python code, and is prettier than your T() function IMO. Steve
participants (7)
-
Anders Hovmöller -
Chris Angelico -
Eric V. Smith -
Hans Polak -
Niki Spahiev -
Stephen J. Turnbull -
Steve Barnes