Mailman 3 January 2017 - Linux-sig

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Nick Coghlan Jan. 7, 2017

Jan. 7, 2017

On 3 January 2017 at 16:00, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > I'm posting it here to ask if anyone sees potential deal-breakers for other > non-Fedora-derived distros before I post it to python-dev for review. Thanks for all the feedback folks, I've now incorporated it into the PEP here: https://github.com/python/peps/commit/221099d8765125bbd798e869846b005bcca84… With the SIG's feedback incorporated, the next round of discussion will take place on python-ideas in parallel with the PEP 540 discussions :) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Nick Coghlan Jan. 5, 2017

Jan. 5, 2017

On 5 January 2017 at 04:08, Robert Collins <robertc(a)robertcollins.net> wrote: > One thing: is the warning on stderr really needed? It's pretty poor form for > the VM to communicate with the user except in extraordinary circumstances > like exception handler of last resort. I think it's appropriate in this case, as the major problem here is that the assumption of ASCII as the preferred text encoding in the "C" locale is an antiquated Anglo-centric default in *C* that has been superceded by UTF-8 or UTF-16-LE in newer languages and runtimes. While I'm not planning to hold my breath, I kinda hope we'll see a change over the next several years where C.ASCII and POSIX.ASCII are introduced as aliases for the current C and POSIX locales, with those two names subsequently transitioning to become aliases for C.UTF-8 (with the latter being reported as the canonical locale name so fallbacks like the one proposed in PEP 538 don't trigger). So what this PEP is doing is taking CPython and saying that we no longer respect the default locale handling behaviour defined by the C language standards, as we think they're wrong (based on our experiences attempting to use them in a multilingual locale dependent application), and so we're overruling them. If people insist on using the default locale anyway, their Python 3 runtime isn't going to work properly, and it isn't a bug in CPython, it's a bug in their environmental configuration. That gives the rationale for the two different warnings: - overruling the default C locale after decades of respecting it (or at least trying to) is a big deal, and hence worth warning about - Python 3's Unicode support is genuinely unreliable in the C locale (everywhere other than Mac OS X), and hence worth warning about > We have verbose mode, and warning on that would make sense to me. Folks that encounter Python 3's deficiencies in the C locale (or hit integration issues arising from the new implicit locale reconfiguration behaviour) without a runtime warning of some kind aren't likely to think "I should run Python in verbose mode to learn more about what's going on", they're more likely to think "Python 3 is broken, I'm going to use something else". Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Robert Collins Jan. 4, 2017

Jan. 4, 2017

One thing: is the warning on stderr really needed? It's pretty poor form for the VM to communicate with the user except in extraordinary circumstances like exception handler of last resort. We have verbose mode, and warning on that would make sense to me. Rob On 3 Jan 2017 7:00 PM, "Nick Coghlan" <ncoghlan(a)gmail.com> wrote: > Hi folks, > > I have a new PEP up for a change I originally suggested as a downstream > patch for Fedora (now that we ship a C.UTF-8 locale by default): > https://www.python.org/dev/peps/pep-0538/ > > I won't post the whole thing here (since a lot of it is background on the > C locale system in general, as well as various things we've tried in the > past), and instead will just summarise the specific technical changes I'm > proposing: > > * in Py_Initialize, emit a warning on stderr regarding limited Unicode > compatibility if we detect that LC_CTYPE is set to the "C" locale > * in Programs/python.c (i.e. the C level main() implementation), set LANG > and LC_ALL in the environment to "C.UTF-8" if we detect that the locale is > otherwise set to "C" > * skip the coercion if PYTHONALLOWCLOCALE is set so developers running in > recent system Python versions with this implemented can still debug > problems that only show up in older Python 3.x releases, or in embedding > applications that still use the C locale > * grant a priori permission to redistributors to backport this to older > versions (as we'd like to include the change in the Fedora system Python > for F26, which will be based on Python 3.6.0) > > I'm posting it here to ask if anyone sees potential deal-breakers for > other non-Fedora-derived distros before I post it to python-dev for review. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia > > _______________________________________________ > Linux-sig mailing list > Linux-sig(a)python.org > https://mail.python.org/mailman/listinfo/linux-sig > >

1 0

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Barry Warsaw Jan. 4, 2017

Jan. 4, 2017

Thanks Nick. I think we're pretty much in agreement on the details, and I'm happy to continue fleshing out the PEP on GH. Just a few comments here. On Jan 04, 2017, at 04:31 PM, Nick Coghlan wrote: >I avoided the term "embedded", as I think it would be confusing when >locale coercion is disabled for the main Python CLI app. > >> It >> also feels odd to provide instructions on how to reproduce this in `python` >> cli from the embedded warning. > >That was a request from some of the Fedora folks, as many of the >developers encountering this warning are expected to be software >maintenance engineers that will want to reproduce integration issues >in a standalone Python runtime. > >However, I agree it reads strangely, and its arguably redundant given >the locale coercion warning when running the main Python CLI app. So >I'll drop it from the upstream PEP, and if we decide we really want it >for the Fedora system Python, we can tweak the wording in a downstream >patch. I like the sample text in #171. >If you missed it, I think I need to better highlight in the PEP that >the library does not, and cannot, coerce the locale to C.UTF-8: >Py_Initialize runs too late in the startup process for that to work >they way we would want it to. Yes, I definitely missed this and I think it's important to emphasize that in the PEP. I think it would be worth adding some text/sample code about how embedders can properly initialize their own applications before they call Py_Initialize(). >But yeah, that's a good idea. Since the runtime initialization warning >and the CLI locale coercion are technically independent, what do you >think about adding two flags: > >* --with[out]-c-locale-coercion (setting PY_COERCE_C_LOCALE for the >CLI behaviour) >* --with[out]-c-locale-warning (setting PY_WARN_ON_C_LOCALE for the >runtime initialization behaviour) This work for me *if* there's really clear documentation about the scope of the effects of these flags, i.e. just as you say above: coercion can only happen in the `python` cli, while the coercion can't happen (and thus only the warning can be issued) in the library/embedded case. >If more people were in the habit of setting sensible locales in their >Docker base images, I doubt I would be bothered about it for Fedora et >al either. Funnily, I ran into a case yesterday where C.UTF-8 *wasn't* set by default on Debian. We have a suite of tools for building packages -well, several, but in this case it's sbuild/schroot- and inside the isolated build environment, C.ASCII is apparently the default. I believe there are open bugs about trying to change this to C.UTF-8 but in any case I was working on a package that failed to build until I explicitly set the locale to C.UTF-8 in the rules file, because the upstream test suite implicitly used ASCII to decode some bytes. So I think this will be useful even for Debian/Ubuntu although we'll likely see lots of these warnings in the package build environment. (Unless, and I haven't checked, the standard Python build helpers already set C.UTF-8; the package in question doesn't use the standard Python helpers.) Cheers, -Barry

1 0

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Barry Warsaw Jan. 4, 2017

Jan. 4, 2017

Hi Nick, thanks for writing up this PEP. I'm generally in favor of it, although as you point out, it will probably have less direct effect on Debian and Ubuntu, since we already have the C.UTF-8 locale. I don't have the time right now to test whether that still holds for various container and remote environments, but I'll try to do some research there (and would actually be surprised if that doesn't hold through the entire food chain). A question and a suggestion. On Jan 03, 2017, at 04:00 PM, Nick Coghlan wrote: >* in Py_Initialize, emit a warning on stderr regarding limited Unicode >compatibility if we detect that LC_CTYPE is set to the "C" locale So just to be clear, you propose only to check for exactly the "C" locale? For example, my default locale is en_US.UTF-8 which would not trigger the warning. I wouldn't want it to warn on any .UTF-8 locale since those should be fine too. (I.e. it's just C locale's implicit ASCII that's the problem.) >* in Programs/python.c (i.e. the C level main() implementation), set LANG >and LC_ALL in the environment to "C.UTF-8" if we detect that the locale is >otherwise set to "C" >* skip the coercion if PYTHONALLOWCLOCALE is set so developers running in >recent system Python versions with this implemented can still debug >problems that only show up in older Python 3.x releases, or in embedding >applications that still use the C locale I have nits to pick about the envar name and warning text. I understand the desire to have a positive setting affect this, but it feels more like PYTHONCOERCECLOCALE=0 would be a more descriptive name and setting. That could be problematic because it doesn't allow any value; i.e. PYTHONCOERCECLOCALE=1 wouldn't make sense to disable locale coercion. I think my unease about the name stems from potential misunderstandings about C vs. C.UTF-8, but maybe I'm just worried about a non-problem. Consider this a challenge for a better envar name... or a bikeshed to ignore. :) On to the warnings: When Py_Initialize is called and CPython detects that the configured locale is the default C locale, the following warning will be issued: Py_Initialize detected LC_CTYPE=C, which limits Unicode compatibility. Some libraries and operating system interfaces may not work correctly. Set `PYTHONALLOWCLOCALE=1 LC_CTYPE=C` to configure a similar environment when running Python directly. I find this confusing on several fronts. I think it might be better to say "Embedded Python" rather than "Py_Initialize" since end users who are using an application with Python embedded probably will have no idea what "Py_Initialize" is, and they are the ones who will see this warning first. It also feels odd to provide instructions on how to reproduce this in `python` cli from the embedded warning. It also doesn't say that the locale is being coerced. What about: Embedded Python detected LC_CTYPE=C (a locale with default ASCII encoding), which may cause Unicode compatibility problems. Coercing the locale to C.UTF-8. Set the environment variable PYTHONALLOWCLOCALE=1 to prevent this coercion. If C.UTF-8 isn't available, then the warning would read: Embedded Python detected LC_CTYPE=C (a locale with default ASCII encoding), which may cause some Unicode compatibility problems. Coercion to C.UTF-8 locale is not possible. Set the environment variable PYTHONALLOWCLOCALE=1 to suppress this warning. I'd use the same text for `python` as cli except s/Embedded Python/Python/ I also think there should be a compile-time or run-time flag that embedders could set so that they could explicitly disable the warning or coercion. Something like ASCIILOCALEISFINEANDYESIKNOWWHATIAMDOINGSOSTFU=1 >* grant a priori permission to redistributors to backport this to older >versions (as we'd like to include the change in the Fedora system Python >for F26, which will be based on Python 3.6.0) I think that's fine, but I doubt we'll need it for Debian and derivatives. Cheers, -Barry

3 4

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Felix Yan Jan. 4, 2017

Jan. 4, 2017

On 01/03/2017 05:43 PM, Nick Coghlan wrote: > If it isn't there, CPython 3.7 will still fall back to the C locale, > same as it does for any other missing locale I see, that's exactly what I wanted to know. > Distros that want to ship Python 3.7, but don't want their users to > see that warning would then need to do one of three things: I am fine with having the warning around, I just wanted to be sure that it won't just break. Thanks for the detailed info! -- Regards, Felix Yan

1 0

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Felix Yan Jan. 3, 2017

Jan. 3, 2017

On 01/03/2017 04:43 PM, Nick Coghlan wrote: > On 3 January 2017 at 17:51, Felix Yan <felixonmars(a)archlinux.org > IMHO it would be nice to have an option to disable the usage of C.UTF-8. > > > Yep, that's part of the PEP - if you set PYTHONALLOWCLOCALE, CPython 3.7 > would still complain about it, but it wouldn't try to coerce the locale > to something else. Since we know that our glibc is not providing C.UTF-8, it would be better to auto-detect the availability of that locale, or simply make it a configure switch, instead of having to set PYTHONALLOWCLOCALE for every python process. -- Regards, Felix Yan

2 1

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Felix Yan Jan. 3, 2017

Jan. 3, 2017

On 01/03/2017 02:00 PM, Nick Coghlan wrote: > I'm posting it here to ask if anyone sees potential deal-breakers for > other non-Fedora-derived distros before I post it to python-dev for review. AFAIK the C.UTF-8 locale is still a downstream patch and not accepted in glibc upstream. Arch has closed the request as wontfix some years ago: https://bugs.archlinux.org/task/32296 IMHO it would be nice to have an option to disable the usage of C.UTF-8. -- Regards, Felix Yan

2 1

PEP 538: Coercing the legacy C locale to C.UTF-8
by Nick Coghlan Jan. 3, 2017

Jan. 3, 2017

Hi folks, I have a new PEP up for a change I originally suggested as a downstream patch for Fedora (now that we ship a C.UTF-8 locale by default): https://www.python.org/dev/peps/pep-0538/ I won't post the whole thing here (since a lot of it is background on the C locale system in general, as well as various things we've tried in the past), and instead will just summarise the specific technical changes I'm proposing: * in Py_Initialize, emit a warning on stderr regarding limited Unicode compatibility if we detect that LC_CTYPE is set to the "C" locale * in Programs/python.c (i.e. the C level main() implementation), set LANG and LC_ALL in the environment to "C.UTF-8" if we detect that the locale is otherwise set to "C" * skip the coercion if PYTHONALLOWCLOCALE is set so developers running in recent system Python versions with this implemented can still debug problems that only show up in older Python 3.x releases, or in embedding applications that still use the C locale * grant a priori permission to redistributors to backport this to older versions (as we'd like to include the change in the Fedora system Python for F26, which will be based on Python 3.6.0) I'm posting it here to ask if anyone sees potential deal-breakers for other non-Fedora-derived distros before I post it to python-dev for review. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0