<div><span style="color: rgb(160, 160, 168); ">On Tuesday, March 5, 2013 at 7:27 AM, Mark McLoughlin wrote:</span></div>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><div>Hi Nick,</div><div><br></div><div>On Tue, 2013-03-05 at 17:56 +1000, Nick Coghlan wrote:</div><blockquote type="cite"><div><div>On Tue, Mar 5, 2013 at 8:29 AM, Mark McLoughlin <<a href="mailto:markmc@redhat.com">markmc@redhat.com</a>> wrote:</div><blockquote type="cite"><div><div>On Mon, 2013-03-04 at 12:44 -0500, Donald Stufft wrote:</div><blockquote type="cite"><div>On Monday, March 4, 2013 at 12:36 PM, Mark McLoughlin wrote:</div></blockquote><div><br></div><blockquote type="cite"><div><blockquote type="cite"><div><div>If parallel incompatible installs is a hopeless problem in Python,</div><div>why</div><div>the push to semantic versioning then rather than saying that</div><div>incompatible API changes should mean a name change?</div></div></blockquote><div>Forcing a name change feels ugly as all hell. I don't really see what</div><div>parallel installs has much to do with anything. I don't bundle anything</div><div>and i'm ideologically opposed to it generally but I don't typically have</div><div>a need for parallel installs because I use virtual environments. Why</div><div>don't you utilize those? (Not being snarky, actually curious).</div></div></blockquote><div><br></div><div>It's a fair question.</div><div><br></div><div>To answer it with a question, how do you imagine Linux distributions</div><div>using virtual environments such that:</div><div><br></div><div> $> yum install -y openstack-nova</div><div><br></div><div>uses a virtual environment? How does it differ from bundling? (Not being</div><div>snarky, actually curious :)</div><div><br></div><div>The approach that some Fedora folks are trying out is called "Software</div><div>Collections". It's not Python specific, but it's basically the same as a</div><div>virtual environment.</div><div><br></div><div>For OpenStack, I think we'd probably have all the Python libraries we</div><div>require installed under e.g. /opt/rh/openstack-$version so that you</div><div>could have programs from two different releases of OpenStack installed</div><div>on the same system.</div><div><br></div><div>Long time packagers are usually horrified at this idea e.g.</div><div><br></div><div> <a href="http://lists.fedoraproject.org/pipermail/devel/2012-December/thread.html#174872">http://lists.fedoraproject.org/pipermail/devel/2012-December/thread.html#174872</a></div></div></blockquote><div><br></div><div>Yes, it's the eternal tension between "I only care about making a wide</div><div>variety of applications on as easy to maintain on platform X as</div><div>possible" view of the sysadmin and the "I only care about making</div><div>application Y as easy to maintain on a wide variety of platforms as</div><div>possible" view of the developer.</div><div><br></div><div>Windows, Android, Mac OS X, etc, pretty much dial their software</div><div>distribution model all the way towards the developer end of the</div><div>spectrum. Linux distro maintainers need to realise that the language</div><div>communities are almost entirely down the developer end of this</div><div>spectrum, where sustainable cross-platform support is much higher</div><div>priority than making life easier for administrators for any given</div><div>platform.</div></div></blockquote><div><br></div><div>I'm with you to there, but it's a bit of a vicious circle - app</div><div>developers bundle to insulate themselves from platform instability and</div><div>then the platform maintainers no longer see a benefit to platform</div><div>stability.</div><div><br></div><blockquote type="cite"><div><div>We're willing to work with distros to make deployment of</div><div>security updates easier, but any proposals that involve people</div><div>voluntarily making cross-platform development harder simpler aren't</div><div>going to be accepted.</div></div></blockquote><div><br></div><div>Let's take it to an extreme and say there was some way to force library</div><div>maintainers to never make incompatible API changes without renaming the</div><div>project. Not what I'm suggesting, of course.</div><div><br></div><div>Doing that would not make app developers lives any/much harder since</div><div>they're bundling anyway.</div><div><br></div><div>It does, however, make life more difficult for the platform maintainers.</div><div><br></div><div>So, what I think you're really saying would be rejected is "any</div><div>proposals which make platform maintenance harder while not providing a</div><div>material benefit to app maintainers who bundle".</div><div><br></div><div>The concrete thing we're discussing here is that distros get screwed if</div><div>the incompatible API changes are commonplace and there is no easy way</div><div>for the distro to ship/install multiple versions of the same API without</div><div>going down the route of every app in the distro bundling their own</div><div>version of the API.</div><div><br></div><div>I don't see why that's a problem that necessarily requires disrupting</div><div>cross-platform app authors in order to address.</div><div><br></div><blockquote type="cite"><div><blockquote type="cite"><div><div> - How many of these 200 new packages are essentially duplicates? Once</div><div> you go down the route of having applications bundle libraries like</div><div> this, there's going to basically be no sharing.</div></div></blockquote><div><br></div><div>There's no sharing only if you *actually* bundle the dependencies into</div><div>each virtualenv. While full bundling is the only mode pip currently</div><div>implements, completely isolating each virtualenv, it doesn't *have* to</div><div>work that way. In particular, PEP 426 offers the opportunity to add a</div><div>"compatible release" mode to pip/virtualenv where the tool can</div><div>maintain a shared pool of installed libraries, and use *.pth files to</div><div>make an appropriate version available in each venv. Updating the</div><div>shared version to a more recent release would then automatically</div><div>update any venvs with a *.pth file that reference that release.</div><div><br></div><div>For example, suppose an application requires "somedep (1.3)". This</div><div>requires at least version 1.3, and won't accept 2.0. The latest</div><div>available qualifying version might be "1.5.3".</div><div><br></div><div>At the moment, pip will install a *copy* of somedep 1.5.3 into the</div><div>application's virtualenv. However, it doesn't have to do that.</div></div></blockquote><div><br></div><div>Awesome! Now we're getting on to figuring out a solution to the</div><div>"parallel installs of multiple incompatible versions" issue :)</div><div><br></div><blockquote type="cite"><div><div>It</div><div>could, instead, install somedep 1.5.3 into a location like</div><div>"/usr/lib/shared/pip-python/somedep1/<contents>", and then add a</div><div>"somedep1.pth" file to the virtualenv that references</div><div>"/usr/lib/shared/pip-python/somedep1/".</div><div><br></div><div>Now, suppose we install another app, also using a virtualenv, that</div><div>requires "somedep (1.6)". The new version 1.6.0 is available now, so</div><div>we install it into the shared location and *both* applications will</div><div>end up using somedep 1.6.0.</div><div><br></div><div>A security update is released for "somedep" as 1.6.1 - we install it</div><div>into the shared location, and now both applications are using 1.6.1</div><div>instead of 1.6.0. Yay, that's what we wanted, just as if we had</div><div>runtime version selection, only the selection happens at install time</div><div>(when adding the *.pth file to the virtualenv) rather than at</div><div>application startup.</div><div><br></div><div>Finally, we install a third application that needs "somedep (2.1)". We</div><div>can't overwrite the shared version, because it isn't compatible.</div><div>Fortunately, what we can do instead is install it to</div><div>"/usr/lib/shared/pip-python/somedep2/<contents>" and create a</div><div>"somedep2.pth" file in that environment. The two virtualenvs relying</div><div>on "somedep1" are blissfully unaware anything has changed because that</div><div>version never appears anywhere on their sys.path.</div><div><br></div><div>Could you use this approach for the actual system site-packages</div><div>directory? No, because sys.path would become insanely long with that</div><div>many *.pth files. However, you could likely symlink to releases stored</div><div>in the *.pth friendly distribution store. But for application specific</div><div>virtual environments, it should be fine.</div></div></blockquote><div><br></div><div>Ok, there's a tonne of details there about pip, virtualenv and .pth</div><div>files that are going over my head right now, but the general idea I'm</div><div>taking away is:</div><div><br></div><div> - the system has multiple versions of somedep installed under /usr </div><div> somewhere</div><div><br></div><div> - the latest version (2.1) is what you get if you naively just do </div><div> 'import somedep'</div><div><br></div><div> - most applications should instead somehow explicitly say they need</div><div> ~>1.3, ->1.6 or ~->2.0 or whatever</div><div><br></div><div> - distros would have multiple copies of the same library, but only </div><div> one copy for each incompatible stream rather than one copy for each </div><div> application</div><div><br></div><div>That's definitely workable.</div><div><br></div><blockquote type="cite"><div><div>If any distros want that kind of thing to become a reality, though,</div><div>they're going to have to step up and contribute it. As noted above,</div><div>for the current tool development teams, the focus is on distributing,</div><div>maintaining and deploying cross-platform applications, not making it</div><div>easy to do security updates on a Linux distro. I believe it's possible</div><div>to satisfy both parties, but it's going to be up to the distros to</div><div>offer a viable plan for meeting their needs without disrupting</div><div>existing upstream practices.</div></div></blockquote><div><br></div><div>Point well taken.</div><div><br></div><div>However, I am surprised the pendulum has swung such that Python platform</div><div>maintainers only worry about app maintainers and not distro maintainers.</div><div><br></div><div>And also, Python embracing incompatible updates and bundling is either a</div><div>new upstream practice or just that doesn't well understood on the distro</div><div>side. </div><div><br></div><blockquote type="cite"><div><div>I will note that making this kind of idea more feasible is one of the</div><div>reasons I am making "compatible release" the *default* in PEP 426</div><div>version specifiers, but it still needs people to actually figure out</div><div>the details and write the code.</div></div></blockquote><div><br></div><div>I still think that going down this road without the parallel installs</div><div>issue solved is a dangerous move for Python. Leaving aside pain for</div><div>distros for a moment, there was a perception (perhaps misguided) that</div><div>Python was a more stable platform that e.g. Ruby or Java.</div><div><br></div><div>If Python library maintainers will see PEP426 as a license to make</div><div>incompatible changes more often so long as they bump their major number,</div><div>then that perception will change.</div></div></div></span></blockquote><div>I still don't really see how this is related to PEP426 unless PEP426 has gotten</div><div>a lot larger since I last looked at it. Where in particular a distribution gets</div><div>installed is left up to the installers to sort out. And making sure that the installed</div><div>versions exist in sys.path is similarly out of scope for PEP426.</div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><div><br></div><blockquote type="cite"><div><div>I will also note that the filesystem layout described above is *far*</div><div>more amenable to safe runtime selection of packages than the current</div><div>pkg_resources method. The critical failure mode in pkg_resources that</div><div>can lead to a lot of weirdness is that it can end up pushing</div><div>site-packages itself on to the front of sys.path which can shadow a</div><div>*lot* of modules (in particular, an installed copy of the software</div><div>you're currently working on may shadow the version in your source</div><div>checkout - this is the bug the patch I linked earlier was needed to</div><div>resolve). Runtime selection would need more work than getting virtual</div><div>environments to work that way, but it's certainly feasible once the</div><div>installation layout is updated.</div></div></blockquote><div><br></div><div>Ok.</div><div><br></div><blockquote type="cite"><div><blockquote type="cite"><div><div> - What's the chance that that all of these 200 packages will be kept</div><div> up to date? If an application works with a given version of a</div><div> library and it can stick with that version, it will. As a Python</div><div> library maintainer, wow do you like the idea of 10 different</div><div> versions of you library included in Fedora?</div></div></blockquote><div><br></div><div>That's a problem the distros need to manage by offering patches to how</div><div>virtual environments and installation layouts work, rather than</div><div>lamenting the fact that cross-platform developers and distro</div><div>maintainers care about different things.</div></div></blockquote><div><br></div><div>I'm not lamenting what cross-platform developers care about. I'm</div><div>lamenting that the Python platform maintainers care more about the</div><div>cross-platform developers than distro maintainers :)</div><div><br></div><blockquote type="cite"><div><blockquote type="cite"><div><div> - The next time a security issue is found in a common Python library,</div><div> does Fedora now have to rush out 10 parallel fixes for it?</div></div></blockquote><div><br></div><div>Not if Fedora contributes the changes needed to support parallel</div><div>installs without requiring changes to existing Python applications and</div><div>libraries.</div></div></blockquote><div><br></div><div>"Patches welcome" - I get it.</div><div><br></div><blockquote type="cite"><div><blockquote type="cite"><div><div>You can see that reaction in mails like this:</div><div><br></div><div> <a href="http://lists.fedoraproject.org/pipermail/devel/2012-December/174944.html">http://lists.fedoraproject.org/pipermail/devel/2012-December/174944.html</a></div><div><br></div><div>and the "why can't these losers just maintain compatibility" view:</div><div><br></div><div> <a href="http://lists.fedoraproject.org/pipermail/devel/2012-December/175028.html">http://lists.fedoraproject.org/pipermail/devel/2012-December/175028.html</a></div><div> <a href="http://lists.fedoraproject.org/pipermail/devel/2012-December/174929.html">http://lists.fedoraproject.org/pipermail/devel/2012-December/174929.html</a></div><div><br></div><div>Notice folks complaining about Ruby and Java here, not Python. I can see</div><div>Python embracing semantic versioning and "just use venv" shortly leading</div><div>to Python being included in the list of "heretics".</div></div></blockquote><div><br></div><div>Unlike Java, the Python community generally sees *actual* bundling as</div><div>evil</div></div></blockquote><div><br></div><div>I think what you call "*actual* bundling" is what I think of as</div><div>"vendorisation" - i.e. where an app actually copies a library into its</div><div>source tree?</div><div><br></div><div>By bundling, I mean that an app sees itself as in control of the</div><div>versions of its dependencies. The app developer fundamentally thinks she</div><div>is delivering a specific stack of dependencies and her application code</div><div>on top rather than installing just their app and running it on a stable</div><div>platform.</div></div></div></span></blockquote><div>In this case the app would be one unit and an upgrade in one of the</div><div>bundled dependencies would require a new version of the App. In my</div><div>mind when an app either bundles or vendorizes they take responsibility</div><div>for those dependencies and making sure they are up to date because</div><div>they are now part of the logical unit that is the app.</div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><div><br></div><blockquote type="cite"><div><div> - expressing constraints relative to a published package index is</div><div>a different thing. Dependencies in Python are typically only brought</div><div>together into a cohesive, pinned set of version by application</div><div>developers and system integrators - the frameworks and libraries often</div><div>express quite loose version requirements (and receive complaints if</div><div>they're overly restrictive).</div><div><br></div><div>The distros just have a harder problem than most because the set of</div><div>packages they're trying to bring together is so large, they're bound</div><div>to run into many cases of packages that have mutually incompatible</div><div>dependencies.</div></div></blockquote><div><br></div><div>It only takes two apps requiring two incompatible versions of the same</div><div>libary for this to become an issue.</div><div><br></div><div>A specific example that concerns OpenStack is you will often want server</div><div>A from version N installed alongside server B from version N+1. This is</div><div>especially true while you're migrating your deployment from version N to</div><div>N+1 since you probably want to upgrade a server at a time.</div><div><br></div><div>Thus, in OpenStack's case, it only takes one of our dependencies to</div><div>release an incompatible version for this to become an issue.</div><div><br></div><div>Python can be a stable platform and OpenStack wouldn't bundle, or it can</div><div>be an unstable platform without parallel installs and OpenStack will</div><div>bundle, or it can be an unstable platform with parallel installs and</div><div>OpenStack won't have to bundle.</div><div><br></div><div>Anyway, sounds like we have some ideas for parallel installs we can</div><div>investigate.</div><div><br></div><div>Thanks,</div><div>Mark.</div></div></div></span>
</blockquote>
<div>
<br>
</div>