[Distutils] Builders vs Installers

[Daniel Holth]

On Mon, Mar 25, 2013 at 7:15 PM, PJ Eby <pje at telecommunity.com> wrote:
> On Mon, Mar 25, 2013 at 5:08 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>> There's a longer-term issue that occurred to me when thinking about
>> pip's role as a "builder" or an "installer" (to use Nick's
>> terminology).
>>
>> As I understand Nick's vision for the future, installers (like pip)
>> will locate built wheels and download and install them, and builders
>> (like distutils and bento) will be responsible for building wheels.
>> But there's an intermediate role which shouldn't get forgotten in the
>> transition - the role that pip currently handles with the "pip wheel"
>> command. This is where I specify a list of distributions, and pip
>> locates sdists, downloads them, checks dependencies, and ultimately
>> builds all of the wheels. I'm not sure whether the current idea of
>> builders includes this "locate, download and resolve dependencies"
>> function (distutils and bento certainly don't have that capability).
>
> Yes, and to make things even more interesting, consider the cases
> where there are build-time dependencies.  ;-)
>
> I would guess that installing from sdists (and revision control) is
> probably here to stay, along with the inherent coupling between
> "build" and "fetch" functions.
>
> Right now, the "build" side of setuptools fetches build-time
> dependencies, but in the New World, ISTM that a top-level build tool
> would just be something that reads the package metadata, finds
> build-time dependencies, and then runs some entry points to ask for a
> wheel to be spit out (or to get back a data structure describing the
> wheel, anyway).
>
> This part could be standardized and indeed could be just something
> that pip does.
>
> Another piece that hasn't seemed well-specified so far is what Nick
> called "archiving" - creating the sdist.  Or perhaps more precisely,
> generating an sdist PKG-INFO and putting the stuff together.
>
> IMO, a good install tool needs to be able to run the archiving and
> building steps as well as installing directly from wheels.  However,
> although metadata 2.0 provides us with a good basis for running build
> and install steps, there really isn't anything yet in the way of a
> standard for sdist generation.
>
> Of course, we have "setup.py sdist", and the previous work on
> setup.cfg by the distutil2 team.  We could either build on setup.cfg,
> or perhaps start over with a simple spec to say how the package is to
> be built: i.e., just a simple set of entry points saying what
> archiver, builder, etc. are used.  Such a file wouldn't change much
> over the life of the package, and would avoid the need for all the
> dynamic hooks provided by distutils2's setup.cfg.
>
> In the degenerate case, I suppose, it could just be "pyarchiver.cfg"
> and contain a couple lines saying what tool is used to generate
> PKG-INFO.
>
> On the other hand, we could draw the line at saying, pip only ever
> installs from sdists, no source checkouts or tarballs.  I'm not sure
> that's a reasonable limitation, though.
>
> On the *other* other hand, Perhaps it would be best to just use the
> setup.cfg work, updated to handle the full metadata 2.0 spec.  As I
> recall, the setup.cfg format handled a ridiculously large number of
> use cases in a very static format, and IIRC still included the
> possibility for dynamic hooks to affect the metadata-generation and
> archive content selection processes, let alone the build and other
> stages.
>
> But then, is that biasing against e.g. bento.info?  Argh.  Packaging
> is hard, let's go shopping.  ;-)
>
> On balance, I think I lean towards just having a simple way to specify
> your chosen archiver, so that installing from source checkouts and
> dumps is possible.  I just find it annoying that you have to have
> *two* files in your checkout, one to say what tool you're using, and
> another one to configure it.
>
> (What'd be nice is if you could just somehow detect files like
> bento.info and setup.cfg and thereby detect what archiver to use.  But
> that would have limited extensibility unless there was a standard
> naming convention for the files, or a standardized format for at least
> the first line in the file or something like that, so you could
> identify the needed tool.)

The problem we are solving first is not "setuptools", it is simply
that right now Python programs are usually installed by downloading
and then running a program that performs the actual install. Instead,
we're going to let the installer do the actual install.

The funny thing is that all the same things happen. We just move the
responsibilities around a little bit, things like Bento or distutils2
don't have to implement their own installer code, and we don't have to
worry about that code doing bad things like messing up our systems or
accessing the Internet. The installer might do new clever things now
that it really controls the install.

There are a tremendous number of things you can do from there, mostly
undeveloped, including decoupling the rest of the packaging pipeline
all the way down to the humble sdist, but we don't really need to
change the pip user interface. All the same things will continue to
happen, just rearranged in a more modular way.


The MEBS "ultimate punt" design is that you would iterate over build
plugins to recognize an sdist. The sdist would be defined as anything
recognized by a plugin. It's probably more practical to at least name
the preferred build system in a very minimal setup.cfg. This is still
a little bit ugly. In a normal "new style" sdist, you may be able to
trust the PKG-INFO file, but when building from source control you
can't, would need to inspect setup.cfg, and ask the build system to
refresh the metadata.