Mailman 3 October 2015 - Linux-sig

Re: [Linux-SIG] PEP 476 and backward incompatibilities for distros
by Barry Warsaw Oct. 9, 2015

Oct. 9, 2015

On Oct 09, 2015, at 04:24 AM, Nick Coghlan wrote: >For upstream, working on PEP 466 persuaded me that the priority needs to be >on providing safe defaults in order to minimise risks for new users, and >for insufficiently paranoid existing users. 2.7 is a special case because there will never be a 2.8. I think it's a different calculus for Python 3. If we hadn't broken backward compatibility in 3.4.3 folks would have eventually gotten the change in 3.5. That doesn't eliminate the cost on developers for the compatibility break, but it does defer it until they do general 3.5 compatibility work. I'm also not convinced that the good PR we might gain from providing users with a more secure default offsets the PR hit we get for breaking compatibility in point releases. It also gives me a vague anti-we're-all-adults feeling. >The burden then falls on commercial redistributors to effectively manage >backwards incompatible changes on behalf of our customers, rather than >expecting upstream to prioritise our interests over those of the rest of >the community. "Effectively managing" could very well mean backing out the incompatible change. But as my larger point goes, that's may not actually be easy to do. If we're going to break backward compatibility in point releases, we need to advertise that loudly, make those changes auditable, and give people an easier way to return to the previous compatible if insecure default. Not all uses of the default are by naive users who need our help to keep all their bases off the intarwebs. It's also not just downstream distros that could be adversely affected by such changes. >Providing a better migration path for cases where commercial support >obligations apply is what the proposed PEP 493 recommendations are about: >https://www.python.org/dev/peps/pep-0493/ The other thing to keep in mind is that distros generally have a team dedicated to security issues. Was this change in response to a CVE? Certainly those teams are better able to judge the implications for the insecure compatible default for users and applications on their platform. I don't envy the work they do to balance those backward compatibility issues against increased security. A change here by upstream is an end-around of those team's responsibilities. Which might still mean it's the right decision for *Python* (and we can argue that on python-dev) but it behooves us to make that very obvious for downstream consumers who come out on the other side of that equation. >This is on redistributors to handle - if we want more or different >documentation in cases like these, then we need to be paying people to >spend more time on representing our interests (and those of our users) in >CPython upstream development, rather than focusing our efforts solely on >integration work for our respective operating systems. No, because as I said it doesn't only affect distros with paid employees. Maybe it affects distros that are purely volunteer run. Maybe it affects people installing Python from source for their own internal usage. Etc. Upstream is who knows that a compatibility break is coming in a point release. We owe it to our users to first, be very careful about that, and second to make sure that information is very well advertised. >The perpetuation of bad default behaviours in the name of backwards >compatibility is a large part of what turned our industry into the insecure >user hostile mess that it is today, though. So, we're going to have to break compatibility at some point to fix it. The question is where and when? Buried in a point release? Wait until the next major release? In documentation, a tech note, or announcement? Cheers, -Barry

2 1

PEP 476 and backward incompatibilities for distros
by Barry Warsaw Oct. 8, 2015

Oct. 8, 2015

And now for something completely different... In Ubuntu, as probably with other distros, we have long-term support (LTS) releases. We have a process called stable release upgrades (SRU) to allow us to systematically update packages in existing, supported releases and we[*] try to keep LTS versions of Python current with upstream. The policy of breaking backward compatibility, even for security issues, can cause us all kinds of pain. I know these types of changes are controversial even upstream, but they are very problematic for distros. There's even talk about disallowing point updates for Python in LTS releases because of this. That does our users a disservice because they do want all the other great bug fixes in point releases. A current example is captured in this bug: https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1500768 which is related to PEP 476 https://www.python.org/dev/peps/pep-0476/ Sadly, I did not follow the discussion on PEP 476 at the time, and I won't argue in this mailing list whether it was a good idea or not. Here I want to talk about making such changes friendlier to downstream consumers such as Ubuntu. The change showed up in Ubuntu 14.04 LTS when Python 3.4.3 was uploaded. This backward incompatible change broke several other packages and user complaints started coming in. We eventually reverted the upload, but not until after lots of people probably upgraded to get it. So now their systems are broken. I know that urllib3/requests upstream was updated to work with this change, but that is problematic for SRUs because we don't want to just backport the latest versions --with new APIs-- of those into the trusty-updates channel. Those new versions may themselves cause cascading incompatibilities. We could try to cherry pick the fixes in those packages, but that's a lot of additional work, and it doesn't help any other breakages caused by PEP 476. Python need to do a better job of not breaking backward compatibility, or making it easy to identify and omit such compatibility breaking changes when pulled in by downstreams. For example, Misc/NEWS says only this in the What's new in 3.4.3. section: - Issue #22417: Verify certificates by default in httplib (PEP 476). That's way too easy to miss, doesn't include any of the backward compatibility break implications, and doesn't really provide much help in narrowing down the commit or commits related to this. The workarounds described in the PEP's Opting Out section isn't really useful either. If we applied this to the Ubuntu version, how would people know what to do to opt back in? These kinds of incompatible changes in Python stable releases give Python a bad reputation, and make folks very wary about just adopting such any maintenance release. That's not a good situation to be in when your users *also* want the latest updates to get other, non breaking fixes. It's a trade-off for sure. It's not necessarily a bad thing to want to improve your user's default security. It *might* be a bad thing to do that in stable releases. It's *definitely* a bad thing to do it in a way that breaks backward compatibility, or is not very clearly marked and easily ignored by distros which need to make different decisions. Cheers, -Barry [*] well Matthias does a considerable amount of the actual work.

2 1

Re: [Linux-SIG] Introducing a Python launcher for *nix?
by Fred Drake Oct. 8, 2015

Oct. 8, 2015

On Wed, Oct 7, 2015 at 7:55 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > The first motive is consistent cross platform invocation - due to the way > Python invocation on Windows is handled, commands like "python3 -m pip" and > "python3.4 -m pip" don't work, and never will, and the differences in file > system layouts exclude the use of absolute paths. Thanks for explaining, Nick! I agree that's a very real group of use-cases, and this sounds like the right kind of approach for that. Since I'm not in those particular trenches (or the Windows miasma), I probably won't have a clue regarding the details of what these users need. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein

1 0

Re: [Linux-SIG] Introducing a Python launcher for *nix?
by Nick Coghlan Oct. 7, 2015

Oct. 7, 2015

On 8 Oct 2015 08:09, "Fred Drake" <fred(a)fdrake.net> wrote: > > On Wed, Oct 7, 2015 at 4:17 PM, Piotr Ozarowski <ozarow(a)gmail.com> wrote: > > If it's about letting users invoke any Python available (is pypy > > included here, BTW?), then why do you need -2 or -3 or worse: -X.Y? > > I'm not sure what the original motivation for this proposal is; it > appears to have been on a different list. The first motive is consistent cross platform invocation - due to the way Python invocation on Windows is handled, commands like "python3 -m pip" and "python3.4 -m pip" don't work, and never will, and the differences in file system layouts exclude the use of absolute paths. The Python launcher for Windows *does* exist, though, so introducing a "py" script on the *nix side of things offers the opportunity to provide a truly cross platform Python experience modelled on the way that works: https://docs.python.org/3/using/windows.html#python-launcher-for-windows The second motive is to make it easy for individual users to switch to an alternative implementation like PyPy as their default interactive shell without impacting shebang lines and other automated invocations. I also want to allow users to make that switch without needing admin rights on their machine. The third motive is simply to make -m invocations shorter ("py -m pip" rather than "python -m pip"). > > The notion that a script is going to figure out which Python is the > right one just seems pretty strange to me. When developing an > application, I know exactly which Python I want to use, and build that > application with that Python to begin with. You're already a professional developer, though, rather than someone learning about the command line and filesystems as part of learning to program in Python. > Any user-facing scripts > get the right sh-bang line to begin with. If I'm building a library, > I'm going to do things with several versions (tests can be driven with > tox, for example), but library users are going to know what version of > Python their application calls for, and they'll use it. No, they don't necessarily know this. If folks don't want to take my word for that, volunteering as a helper at a Software Carpentry bootcamp, a Python for Young Coders workshop or another community event that introduces children or adults to Python programming highlights just how radically wrong these kinds of assumptions are in the context of people learning to program, especially when instructors are needing to cope with a mixed audience of Windows, OS X, and Linux users. > I may be missing something pretty obvious to many, though. My > approaches to building an application don't appear to be commonly > applied in Python environments. Yeah, I failed to convey that the main intended beneficiaries are future newcomers to Python, and the folks that are themselves using Linux or OS X, but need to provide instructions that also work on Windows. Cheers, Nick. > > > -Fred > > -- > Fred L. Drake, Jr. <fred at fdrake.net> > "A storm broke loose in my mind." --Albert Einstein > _______________________________________________ > Linux-sig mailing list > Linux-sig(a)python.org > https://mail.python.org/mailman/listinfo/linux-sig

1 0

Re: [Linux-SIG] Introducing a Python launcher for *nix?
by Barry Warsaw Oct. 7, 2015

Oct. 7, 2015

[Quoting out of order] >P.S. This list may not have enough subscribers yet to have a good >discussion, so I'll post another thread about advertising, and then we >can reboot this thread later if needed. Agreed. On Oct 07, 2015, at 01:28 PM, Nick Coghlan wrote: >What do folks think of the idea of introducing a "Python Launcher for >*nix" using the following iterative development model: > >1. We update the Python 3 *nix builds to provide a "py" symlink by >default. This would be the baseline state for Python 3.6 if the next >step isn't completed in time. > >2. The symlink is replaced with a custom launcher that works like the >Python 3.5 version of the Windows launcher: > >* accepts "-3", "-2", etc command line arguments like the Windows launcher >* checks for an active virtualenv, if it finds one, uses the venv's >default Python >* checks for a user specific config file >* checks for a global config file I like it, although I would also like to support major.minor specifications. For example, in Ubuntu 15.10 we'll have two supported Python 3 versions, 3.4 and 3.5. We do have /usr/bin/python3 pointing to the default version, i.e. /usr/bin/python3.4 and that changes as we roll over to the new default version (it'll be 3.5 only for the next LTS). So `py -3` is unambiguous but folks might want to use `py -3.4` which kind of looks ugly. Maybe the launcher should take an optional mutually exclusive argument such as -i/--interpreter which would either give a shorthand, like 3.4 or a full path to the interpreter to choose. That would let people install say 3.6 in /usr/local and then they could `py -i /usr/local/bin/python3.6` (Is that a win over just using /usr/local/bin/python3.6? Not really, but I can see the launcher being scriptable so you'd want to allow such scripts to substitute the exact interpreter they want to use. OTOH YAGNI.) Cheers, -Barry

3 2

Re: [Linux-SIG] Introducing a Python launcher for *nix?
by Barry Warsaw Oct. 7, 2015

Oct. 7, 2015

[Reposting for the Gmane newsgroup] [Quoting out of order] >P.S. This list may not have enough subscribers yet to have a good >discussion, so I'll post another thread about advertising, and then we >can reboot this thread later if needed. Agreed. On Oct 07, 2015, at 01:28 PM, Nick Coghlan wrote: >What do folks think of the idea of introducing a "Python Launcher for >*nix" using the following iterative development model: > >1. We update the Python 3 *nix builds to provide a "py" symlink by >default. This would be the baseline state for Python 3.6 if the next >step isn't completed in time. > >2. The symlink is replaced with a custom launcher that works like the >Python 3.5 version of the Windows launcher: > >* accepts "-3", "-2", etc command line arguments like the Windows launcher >* checks for an active virtualenv, if it finds one, uses the venv's >default Python >* checks for a user specific config file >* checks for a global config file I like it, although I would also like to support major.minor specifications. For example, in Ubuntu 15.10 we'll have two supported Python 3 versions, 3.4 and 3.5. We do have /usr/bin/python3 pointing to the default version, i.e. /usr/bin/python3.4 and that changes as we roll over to the new default version (it'll be 3.5 only for the next LTS). So `py -3` is unambiguous but folks might want to use `py -3.4` which kind of looks ugly. Maybe the launcher should take an optional mutually exclusive argument such as -i/--interpreter which would either give a shorthand, like 3.4 or a full path to the interpreter to choose. That would let people install say 3.6 in /usr/local and then they could `py -i /usr/local/bin/python3.6` (Is that a win over just using /usr/local/bin/python3.6? Not really, but I can see the launcher being scriptable so you'd want to allow such scripts to substitute the exact interpreter they want to use. OTOH YAGNI.) Cheers, -Barry

1 0

Re: [Linux-SIG] Announcing/advertising the Python Linux SIG
by Barry Warsaw Oct. 7, 2015

Oct. 7, 2015

On Oct 07, 2015, at 01:38 PM, Nick Coghlan wrote: >Where should we advertise for potential SIG members? I just posted a message to the debian-python@ mailing list. I will post something to ubuntu-devel@ momentarily. FWIW, I also just requested gmane.comp.python.linux-sig All of your other suggestions sound good too. Go for it! Cheers, -Barry

1 0

Announcing/advertising the Python Linux SIG
by Nick Coghlan Oct. 7, 2015

Oct. 7, 2015

Hi folks, Where should we advertise for potential SIG members? My suggestions for the Python community side: * python-dev * python-ideas * distutils-sig * conda mailing list [1] And for the slice of downstream of most interest to me personally: * Fedora Python SIG [2] * Red Hat's internal Python list * softwarecollections.org mailing list [3] * Project Atomic mailing list [4] [1] https://groups.google.com/a/continuum.io/forum/#!forum/conda [2] https://lists.fedoraproject.org/mailman/listinfo/python-devel [3] https://www.redhat.com/mailman/listinfo/sclorg [4] https://lists.projectatomic.io/mailman/listinfo/atomic-devel -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Introducing a Python launcher for *nix?
by Nick Coghlan Oct. 7, 2015

Oct. 7, 2015

The Python Launcher for Windows is a nice feature, but it's difficult for folks that use Linux or Mac OS X themselves to recommend it, as the "py" shorthand only works on Windows. What do folks think of the idea of introducing a "Python Launcher for *nix" using the following iterative development model: 1. We update the Python 3 *nix builds to provide a "py" symlink by default. This would be the baseline state for Python 3.6 if the next step isn't completed in time. 2. The symlink is replaced with a custom launcher that works like the Python 3.5 version of the Windows launcher: * accepts "-3", "-2", etc command line arguments like the Windows launcher * checks for an active virtualenv, if it finds one, uses the venv's default Python * checks for a user specific config file * checks for a global config file (This could potentially be achieved by making the existing launcher cross-platform, rather than having two independent launchers to maintain) Cheers, Nick. P.S. This list may not have enough subscribers yet to have a good discussion, so I'll post another thread about advertising, and then we can reboot this thread later if needed. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0