[Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout

[Nathaniel Smith]

On Aug 20, 2016 11:00 AM, "Jim Fulton" <jim at jimfulton.info> wrote:
>
[...]
> With buildout, I chose to use eggs differently.  I simply generate
scripts with dependencies explicitly listed in the Python path.  This is
very easy to reason about.  It makes it easy to see what's installed by
looking at the generated path.  This is similar to the way Java uses class
paths to assemble jar files, which is fitting given that the design of eggs
took at least some inspiration from jar files.
>
> I'm not a fan of Java, but I think class paths + jar files was something
it got right.  Each program has it's own distinct environment.  If you need
to add/remove/update a dependency, just change the path.  Want to share
installed components between programs?  It's easy because only one
installed component needs to be stored and different programs point to it
via their paths.

Wheels are a pretty simple and straightforward format. They've got some
metadata, and then the are a set of directories with labels attached: "this
directory needs to go on sys.path", "this directory has scripts that should
have shebang fixups applied and then go on $PATH", etc. (The spec is not
long: https://www.python.org/dev/peps/pep-0427/)

My first guess at how buildout could handle wheels elegantly would be:

- we have one piece of code that abstracts the operation of "please give me
a wheel for package X version Y that works on this system". This then
contacts pypi, builds sdists if it has to, whatever. It could be in the
form of an invocation of pip, or it could be a call into some common
library that gets factored out of pip and used by both pip and buildout,
whatever.

- pip uses this code to get a wheel, and once it has a wheel, it installs
it into an environment in the same way it does today.

- buildout *also* uses this code to get a wheel, and once it has a wheel,
it does its own thing: maybe its own dependency resolution, and definitely
its own install strategy, where it unpacks the wheel's directories into its
own preferred layout, does sys.path munging on the wheel's scripts, etc.

AFAICT there's nothing in the wheel format that ties you to the
virtualenv-style directory layout.

The disadvantage is that someone would have to do the work to factor out
that get-a-wheel code, and to write/maintain custom wheel installation code
in buildout, instead of just invoking setuptools. The advantage would be
the now you aren't dependent on setuptools...

> The buildout developers have discussed options for the future. We know
there's a reckoning coming, but so far, thankfully, we've been able to put
it off, but we don't want to be a burden on the rest of the Python
community. (Seriously, thank you for not breaking us. :) )

As a heads up, there is unfortunately another source of breakage heading
down the pipe: the PEP 516/517 plans for making it easy to use
non-distutils-based build systems are heavily tied to wheels, so if/when
the spec gets finished and projects start adopting alternative build
systems, you may see an increasing proliferation of projects that cannot
build eggs or be easy_install'ed.

(Ref: https://www.python.org/dev/peps/pep-0517/)

It's not clear when this will happen exactly -- I'm in the midst of dealing
with some medical stuff, so haven't had much/any time for distutils-sig
stuff recently, but sooner or later this is the direction we want to go.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/020b3fbb/attachment.html>