Mailman 3 - Linux-sig

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Barry Warsaw 04 Jan '17

04 Jan '17

Thanks Nick. I think we're pretty much in agreement on the details, and I'm happy to continue fleshing out the PEP on GH. Just a few comments here. On Jan 04, 2017, at 04:31 PM, Nick Coghlan wrote: >I avoided the term "embedded", as I think it would be confusing when >locale coercion is disabled for the main Python CLI app. > >> It >> also feels odd to provide instructions on how to reproduce this in `python` >> cli from the embedded warning. > >That was a request from some of the Fedora folks, as many of the >developers encountering this warning are expected to be software >maintenance engineers that will want to reproduce integration issues >in a standalone Python runtime. > >However, I agree it reads strangely, and its arguably redundant given >the locale coercion warning when running the main Python CLI app. So >I'll drop it from the upstream PEP, and if we decide we really want it >for the Fedora system Python, we can tweak the wording in a downstream >patch. I like the sample text in #171. >If you missed it, I think I need to better highlight in the PEP that >the library does not, and cannot, coerce the locale to C.UTF-8: >Py_Initialize runs too late in the startup process for that to work >they way we would want it to. Yes, I definitely missed this and I think it's important to emphasize that in the PEP. I think it would be worth adding some text/sample code about how embedders can properly initialize their own applications before they call Py_Initialize(). >But yeah, that's a good idea. Since the runtime initialization warning >and the CLI locale coercion are technically independent, what do you >think about adding two flags: > >* --with[out]-c-locale-coercion (setting PY_COERCE_C_LOCALE for the >CLI behaviour) >* --with[out]-c-locale-warning (setting PY_WARN_ON_C_LOCALE for the >runtime initialization behaviour) This work for me *if* there's really clear documentation about the scope of the effects of these flags, i.e. just as you say above: coercion can only happen in the `python` cli, while the coercion can't happen (and thus only the warning can be issued) in the library/embedded case. >If more people were in the habit of setting sensible locales in their >Docker base images, I doubt I would be bothered about it for Fedora et >al either. Funnily, I ran into a case yesterday where C.UTF-8 *wasn't* set by default on Debian. We have a suite of tools for building packages -well, several, but in this case it's sbuild/schroot- and inside the isolated build environment, C.ASCII is apparently the default. I believe there are open bugs about trying to change this to C.UTF-8 but in any case I was working on a package that failed to build until I explicitly set the locale to C.UTF-8 in the rules file, because the upstream test suite implicitly used ASCII to decode some bytes. So I think this will be useful even for Debian/Ubuntu although we'll likely see lots of these warnings in the package build environment. (Unless, and I haven't checked, the standard Python build helpers already set C.UTF-8; the package in question doesn't use the standard Python helpers.) Cheers, -Barry

1 0

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Barry Warsaw 04 Jan '17

04 Jan '17

Hi Nick, thanks for writing up this PEP. I'm generally in favor of it, although as you point out, it will probably have less direct effect on Debian and Ubuntu, since we already have the C.UTF-8 locale. I don't have the time right now to test whether that still holds for various container and remote environments, but I'll try to do some research there (and would actually be surprised if that doesn't hold through the entire food chain). A question and a suggestion. On Jan 03, 2017, at 04:00 PM, Nick Coghlan wrote: >* in Py_Initialize, emit a warning on stderr regarding limited Unicode >compatibility if we detect that LC_CTYPE is set to the "C" locale So just to be clear, you propose only to check for exactly the "C" locale? For example, my default locale is en_US.UTF-8 which would not trigger the warning. I wouldn't want it to warn on any .UTF-8 locale since those should be fine too. (I.e. it's just C locale's implicit ASCII that's the problem.) >* in Programs/python.c (i.e. the C level main() implementation), set LANG >and LC_ALL in the environment to "C.UTF-8" if we detect that the locale is >otherwise set to "C" >* skip the coercion if PYTHONALLOWCLOCALE is set so developers running in >recent system Python versions with this implemented can still debug >problems that only show up in older Python 3.x releases, or in embedding >applications that still use the C locale I have nits to pick about the envar name and warning text. I understand the desire to have a positive setting affect this, but it feels more like PYTHONCOERCECLOCALE=0 would be a more descriptive name and setting. That could be problematic because it doesn't allow any value; i.e. PYTHONCOERCECLOCALE=1 wouldn't make sense to disable locale coercion. I think my unease about the name stems from potential misunderstandings about C vs. C.UTF-8, but maybe I'm just worried about a non-problem. Consider this a challenge for a better envar name... or a bikeshed to ignore. :) On to the warnings: When Py_Initialize is called and CPython detects that the configured locale is the default C locale, the following warning will be issued: Py_Initialize detected LC_CTYPE=C, which limits Unicode compatibility. Some libraries and operating system interfaces may not work correctly. Set `PYTHONALLOWCLOCALE=1 LC_CTYPE=C` to configure a similar environment when running Python directly. I find this confusing on several fronts. I think it might be better to say "Embedded Python" rather than "Py_Initialize" since end users who are using an application with Python embedded probably will have no idea what "Py_Initialize" is, and they are the ones who will see this warning first. It also feels odd to provide instructions on how to reproduce this in `python` cli from the embedded warning. It also doesn't say that the locale is being coerced. What about: Embedded Python detected LC_CTYPE=C (a locale with default ASCII encoding), which may cause Unicode compatibility problems. Coercing the locale to C.UTF-8. Set the environment variable PYTHONALLOWCLOCALE=1 to prevent this coercion. If C.UTF-8 isn't available, then the warning would read: Embedded Python detected LC_CTYPE=C (a locale with default ASCII encoding), which may cause some Unicode compatibility problems. Coercion to C.UTF-8 locale is not possible. Set the environment variable PYTHONALLOWCLOCALE=1 to suppress this warning. I'd use the same text for `python` as cli except s/Embedded Python/Python/ I also think there should be a compile-time or run-time flag that embedders could set so that they could explicitly disable the warning or coercion. Something like ASCIILOCALEISFINEANDYESIKNOWWHATIAMDOINGSOSTFU=1 >* grant a priori permission to redistributors to backport this to older >versions (as we'd like to include the change in the Fedora system Python >for F26, which will be based on Python 3.6.0) I think that's fine, but I doubt we'll need it for Debian and derivatives. Cheers, -Barry

3 4

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Felix Yan 04 Jan '17

04 Jan '17

On 01/03/2017 05:43 PM, Nick Coghlan wrote: > If it isn't there, CPython 3.7 will still fall back to the C locale, > same as it does for any other missing locale I see, that's exactly what I wanted to know. > Distros that want to ship Python 3.7, but don't want their users to > see that warning would then need to do one of three things: I am fine with having the warning around, I just wanted to be sure that it won't just break. Thanks for the detailed info! -- Regards, Felix Yan

1 0

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Felix Yan 03 Jan '17

03 Jan '17

On 01/03/2017 04:43 PM, Nick Coghlan wrote: > On 3 January 2017 at 17:51, Felix Yan <felixonmars(a)archlinux.org > IMHO it would be nice to have an option to disable the usage of C.UTF-8. > > > Yep, that's part of the PEP - if you set PYTHONALLOWCLOCALE, CPython 3.7 > would still complain about it, but it wouldn't try to coerce the locale > to something else. Since we know that our glibc is not providing C.UTF-8, it would be better to auto-detect the availability of that locale, or simply make it a configure switch, instead of having to set PYTHONALLOWCLOCALE for every python process. -- Regards, Felix Yan

2 1

Re: [Linux-SIG] PEP 538: Coercing the legacy C locale to C.UTF-8
by Felix Yan 03 Jan '17

03 Jan '17

On 01/03/2017 02:00 PM, Nick Coghlan wrote: > I'm posting it here to ask if anyone sees potential deal-breakers for > other non-Fedora-derived distros before I post it to python-dev for review. AFAIK the C.UTF-8 locale is still a downstream patch and not accepted in glibc upstream. Arch has closed the request as wontfix some years ago: https://bugs.archlinux.org/task/32296 IMHO it would be nice to have an option to disable the usage of C.UTF-8. -- Regards, Felix Yan

2 1

PEP 538: Coercing the legacy C locale to C.UTF-8
by Nick Coghlan 03 Jan '17

03 Jan '17

Hi folks, I have a new PEP up for a change I originally suggested as a downstream patch for Fedora (now that we ship a C.UTF-8 locale by default): https://www.python.org/dev/peps/pep-0538/ I won't post the whole thing here (since a lot of it is background on the C locale system in general, as well as various things we've tried in the past), and instead will just summarise the specific technical changes I'm proposing: * in Py_Initialize, emit a warning on stderr regarding limited Unicode compatibility if we detect that LC_CTYPE is set to the "C" locale * in Programs/python.c (i.e. the C level main() implementation), set LANG and LC_ALL in the environment to "C.UTF-8" if we detect that the locale is otherwise set to "C" * skip the coercion if PYTHONALLOWCLOCALE is set so developers running in recent system Python versions with this implemented can still debug problems that only show up in older Python 3.x releases, or in embedding applications that still use the C locale * grant a priori permission to redistributors to backport this to older versions (as we'd like to include the change in the Fedora system Python for F26, which will be based on Python 3.6.0) I'm posting it here to ask if anyone sees potential deal-breakers for other non-Fedora-derived distros before I post it to python-dev for review. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Linux-SIG] Introducing a Python launcher for *nix?
by Wolfgang Maier 12 Nov '15

12 Nov '15

On 08.10.2015 01:55, Nick Coghlan wrote: > > The first motive is consistent cross platform invocation - due to the > way Python invocation on Windows is handled, commands like "python3 -m > pip" and "python3.4 -m pip" don't work, and never will, and the > differences in file system layouts exclude the use of absolute paths. > > The Python launcher for Windows *does* exist, though, so introducing a > "py" script on the *nix side of things offers the opportunity to provide > a truly cross platform Python experience modelled on the way that works: > https://docs.python.org/3/using/windows.html#python-launcher-for-windows > Then this *nix py launcher should also, like its Windows counterpart inspect .py files for presence of a shebang line and call the right interpreter based on it. Of course, on *nix systems you could make the script executable and let the shell determine the interpreter for you, but that's not cross platform compatible invocation then. In essence, you'd want everything that the Windows launcher does except for searching the registry, of course, which AFAIU you suggest replacing with user-specific and global config files, right? Best, Wolfgang

1 0

Re: [Linux-SIG] PEP 476 and backward incompatibilities for distros
by Barry Warsaw 09 Oct '15

09 Oct '15

On Oct 09, 2015, at 04:24 AM, Nick Coghlan wrote: >For upstream, working on PEP 466 persuaded me that the priority needs to be >on providing safe defaults in order to minimise risks for new users, and >for insufficiently paranoid existing users. 2.7 is a special case because there will never be a 2.8. I think it's a different calculus for Python 3. If we hadn't broken backward compatibility in 3.4.3 folks would have eventually gotten the change in 3.5. That doesn't eliminate the cost on developers for the compatibility break, but it does defer it until they do general 3.5 compatibility work. I'm also not convinced that the good PR we might gain from providing users with a more secure default offsets the PR hit we get for breaking compatibility in point releases. It also gives me a vague anti-we're-all-adults feeling. >The burden then falls on commercial redistributors to effectively manage >backwards incompatible changes on behalf of our customers, rather than >expecting upstream to prioritise our interests over those of the rest of >the community. "Effectively managing" could very well mean backing out the incompatible change. But as my larger point goes, that's may not actually be easy to do. If we're going to break backward compatibility in point releases, we need to advertise that loudly, make those changes auditable, and give people an easier way to return to the previous compatible if insecure default. Not all uses of the default are by naive users who need our help to keep all their bases off the intarwebs. It's also not just downstream distros that could be adversely affected by such changes. >Providing a better migration path for cases where commercial support >obligations apply is what the proposed PEP 493 recommendations are about: >https://www.python.org/dev/peps/pep-0493/ The other thing to keep in mind is that distros generally have a team dedicated to security issues. Was this change in response to a CVE? Certainly those teams are better able to judge the implications for the insecure compatible default for users and applications on their platform. I don't envy the work they do to balance those backward compatibility issues against increased security. A change here by upstream is an end-around of those team's responsibilities. Which might still mean it's the right decision for *Python* (and we can argue that on python-dev) but it behooves us to make that very obvious for downstream consumers who come out on the other side of that equation. >This is on redistributors to handle - if we want more or different >documentation in cases like these, then we need to be paying people to >spend more time on representing our interests (and those of our users) in >CPython upstream development, rather than focusing our efforts solely on >integration work for our respective operating systems. No, because as I said it doesn't only affect distros with paid employees. Maybe it affects distros that are purely volunteer run. Maybe it affects people installing Python from source for their own internal usage. Etc. Upstream is who knows that a compatibility break is coming in a point release. We owe it to our users to first, be very careful about that, and second to make sure that information is very well advertised. >The perpetuation of bad default behaviours in the name of backwards >compatibility is a large part of what turned our industry into the insecure >user hostile mess that it is today, though. So, we're going to have to break compatibility at some point to fix it. The question is where and when? Buried in a point release? Wait until the next major release? In documentation, a tech note, or announcement? Cheers, -Barry

2 1

PEP 476 and backward incompatibilities for distros
by Barry Warsaw 08 Oct '15

08 Oct '15

And now for something completely different... In Ubuntu, as probably with other distros, we have long-term support (LTS) releases. We have a process called stable release upgrades (SRU) to allow us to systematically update packages in existing, supported releases and we[*] try to keep LTS versions of Python current with upstream. The policy of breaking backward compatibility, even for security issues, can cause us all kinds of pain. I know these types of changes are controversial even upstream, but they are very problematic for distros. There's even talk about disallowing point updates for Python in LTS releases because of this. That does our users a disservice because they do want all the other great bug fixes in point releases. A current example is captured in this bug: https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1500768 which is related to PEP 476 https://www.python.org/dev/peps/pep-0476/ Sadly, I did not follow the discussion on PEP 476 at the time, and I won't argue in this mailing list whether it was a good idea or not. Here I want to talk about making such changes friendlier to downstream consumers such as Ubuntu. The change showed up in Ubuntu 14.04 LTS when Python 3.4.3 was uploaded. This backward incompatible change broke several other packages and user complaints started coming in. We eventually reverted the upload, but not until after lots of people probably upgraded to get it. So now their systems are broken. I know that urllib3/requests upstream was updated to work with this change, but that is problematic for SRUs because we don't want to just backport the latest versions --with new APIs-- of those into the trusty-updates channel. Those new versions may themselves cause cascading incompatibilities. We could try to cherry pick the fixes in those packages, but that's a lot of additional work, and it doesn't help any other breakages caused by PEP 476. Python need to do a better job of not breaking backward compatibility, or making it easy to identify and omit such compatibility breaking changes when pulled in by downstreams. For example, Misc/NEWS says only this in the What's new in 3.4.3. section: - Issue #22417: Verify certificates by default in httplib (PEP 476). That's way too easy to miss, doesn't include any of the backward compatibility break implications, and doesn't really provide much help in narrowing down the commit or commits related to this. The workarounds described in the PEP's Opting Out section isn't really useful either. If we applied this to the Ubuntu version, how would people know what to do to opt back in? These kinds of incompatible changes in Python stable releases give Python a bad reputation, and make folks very wary about just adopting such any maintenance release. That's not a good situation to be in when your users *also* want the latest updates to get other, non breaking fixes. It's a trade-off for sure. It's not necessarily a bad thing to want to improve your user's default security. It *might* be a bad thing to do that in stable releases. It's *definitely* a bad thing to do it in a way that breaks backward compatibility, or is not very clearly marked and easily ignored by distros which need to make different decisions. Cheers, -Barry [*] well Matthias does a considerable amount of the actual work.

2 1

Re: [Linux-SIG] Introducing a Python launcher for *nix?
by Fred Drake 08 Oct '15

08 Oct '15

On Wed, Oct 7, 2015 at 7:55 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > The first motive is consistent cross platform invocation - due to the way > Python invocation on Windows is handled, commands like "python3 -m pip" and > "python3.4 -m pip" don't work, and never will, and the differences in file > system layouts exclude the use of absolute paths. Thanks for explaining, Nick! I agree that's a very real group of use-cases, and this sounds like the right kind of approach for that. Since I'm not in those particular trenches (or the Windows miasma), I probably won't have a clue regarding the details of what these users need. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein

1 0