[Distutils] Builders vs Installers

PJ Eby pje at telecommunity.com
Wed Mar 27 23:41:45 CET 2013

On Wed, Mar 27, 2013 at 1:51 PM, Donald Stufft <donald at stufft.io> wrote:
> I don't think the packaging formats should dictate the development flow at all. .IN files and such all dictate how that should be.

You can't *not* dictate the flow.  If you don't have something to
generate the file with, then you're dictating that developers must
ship an sdist and users must manually run build tools to make wheels.

(Or, more likely, you're creating the motivation for somebody to
create a meta-meta-build system that solves the problem, probably by
having a bunch of plugins to detect what build system a raw source
directory is using.)

> To me this is an installer issue not a packaging issue and it's best solved in the installers. Obviously there is some benefit to a "standard" way for installers to treat these but I don't think it should be defined in terms of the packaging formats.

It definitely doesn't have to be.  distutils2's setup.cfg isn't
actually a bad human-readable format, but it's not a *packaging*

In any case, the only thing an installer needs is a way to get the
setup-requires-dist, or the portion of it that pertains to identifying
the metadata hooks. The rest could be handled with entry points
registered for configuration file names.  For example, Bento could
expose an entry point like:

  bento.info = some.module.in.bento:hookfunction

And then an installer builds a list of these hooks that are in the
setup-requires-dists, and runs them based on the filenames found in
the project directory.  All done.

Basically, the only thing we need is a way to avoid having to either:

1. Make every user install Bento/etc. before trying to install a
package from source that uses it, or
2. Embed a registry of every possible build configuration file name
into every installer.

And this can be done in any of several ways:

* Have a standardized naming pattern like pybuild.*, and make the .*
part indicate the build tool

* Have a standardized single name (like pybuild.cfg), and encode the
build tool in a string that can be read regardless of file format, so
it can be embedded in whatever format the build tool itself uses

* Have a separate file that *only* lists the build tool or
setup-requires-dists (and maybe can be extended to contain other
information for use with a stdlib-supplied build tool)

I personally lean towards the last one, especially if it reuses
setup.cfg, because setup.cfg already exists and is fairly
standardized.  There are even tools that work today to let you do a
metadata-free setup.py and specify everything needed in setup.cfg,
with environment markers and everything.

Heck, IIUC, there's at least one library you can use today with
*setuptools* to do that -- it doesn't need distutils2 or any of that,
it just translates setup.cfg to setup.py arguments.

But an even more important reason to standardize is that there should
be one, and preferably only one, obvious way to do it.  AFAIK, the
distutils2 effort didn't fail because of setup.cfg -- heck, setup.cfg
was the main *benefit* I saw in the distutils2 work, everything else
about it AFAIK was just setuptools warmed over -- it failed because of
trying to boil the ocean and *implement* everything, rather than just
standardizing on interfaces.  A minimal boilerplate setup.cfg could be
something like

builder = bento >1.6

And leave it at that.  Disadvantage is that it's a dumb boilerplate
file for tools that don't use setup.cfg for their configuration --
i.e., it's a minor disadvantage to users of those tools.  However, if
your preferred build tool generates the file for you, it's no big
deal, as long as the generated file doesn't change all the time and
you check it into source control.

Such a usage pattern is teachable and provides what's needed, without
dictating anything about the development workflow, other than that you
need to tell installers how to make an sdist if you want people to
install stuff you shipped without an sdist or a wheel, or if you want
to use any generic build-running tools that need to know your build

> development checkouts should be discouraged unless you're actively working on that project.

Perhaps Jim can chime in on this point, but when you work with a whole
bunch of people developing a whole bunch of libraries making up a
larger project (e.g. Zope), it doesn't seem very sensible to expect
that everybody manually check out and manage all the dependencies
they're using.  Maybe you could mitigate that somewhat with some sort
automated continuous build/release system, but that's not always a
practical option.

More information about the Distutils-SIG mailing list