[Distutils] Is build an inherently arbitrary-code process?

[David Cournapeau]

On Thu, Mar 27, 2014 at 7:48 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Mar 27, 2014, at 3:42 PM, Daniel Holth <dholth at gmail.com> wrote:
>
> > I became convinced that build was an inherently arbitrary-code
> > process, and not something to be universally handled by a declarative
> > system, when I observed an autotools project under configuration. The
> > things spend ten minutes writing and compiling snippets of C code to
> > determine which features are and are not supported by the runtime.
> > This is *very* arbitrary code and probably one of the tamer things
> > people like to do during builds.
> >
> > As usual I blame distutils, because the separation is not there. So
> > when you blame setup.py for having to run procedural code to generate
> > the *metadata* you might go to far and think you should also eliminate
> > a procedural build - simply because build and metadata are not
> > adequately separate in the distutils design. Declarative build systems
> > are a nice idea but they are not going to work for everyone.
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
> Well for the record auto tools does all of that because there’s near
> zero ability to introspect the compiler to figure out what it supports
> except by compiling little programs that depend on X feature and
> then running them to see if they work or not.
>
> That being said, yes building code is inherently a process that needs
> to execute things and cannot be made completely static. I believe
> the metadata itself can be static, and instead of an arbitrary build
> script we can declare a build hook, but that’s an argument for a PEP!
>
> I do believe a declarative build system can work for the 90% case
> though and should probably be the “default” option.
>

It is indeed possible for most python packages, and I tried to figure out
quite a few things to make that percentage as high as possible in Bento.

The really difficult part is not that, though, but making the system
pluggable to allow pretty much arbitrary extensibility. What makes
distutils very ill suited to that task is the whole subcommand model where
options are not known in advance (that was the #1 issue when integrating
distutils and scons).

For bento, I came up with a system of 'build manifest', where each command
produces a text-based 'protocol' that other commands might use (
https://github.com/pv/bento/blob/master/doc/source/hacking.rst). The
implementation is not great but I think the idea is still sound: this means
that in theory, you could build C extensions with something not involving
python at all (e.g. make), but as long as the build process output that
file, you could build installers from the system. IOW, the json file acts
as 'middleware'.

What any future solution should *not* try to do is inventing yet another
build system

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140327/29e65c6f/attachment.html>