On Thu, Mar 27, 2014 at 7:48 PM, Donald Stufft <donald@stufft.io> wrote:

On Mar 27, 2014, at 3:42 PM, Daniel Holth <dholth@gmail.com> wrote:

> I became convinced that build was an inherently arbitrary-code
> process, and not something to be universally handled by a declarative
> system, when I observed an autotools project under configuration. The
> things spend ten minutes writing and compiling snippets of C code to
> determine which features are and are not supported by the runtime.
> This is *very* arbitrary code and probably one of the tamer things
> people like to do during builds.
>
> As usual I blame distutils, because the separation is not there. So
> when you blame setup.py for having to run procedural code to generate
> the *metadata* you might go to far and think you should also eliminate
> a procedural build - simply because build and metadata are not
> adequately separate in the distutils design. Declarative build systems
> are a nice idea but they are not going to work for everyone.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

Well for the record auto tools does all of that because there’s near
zero ability to introspect the compiler to figure out what it supports
except by compiling little programs that depend on X feature and
then running them to see if they work or not.

That being said, yes building code is inherently a process that needs
to execute things and cannot be made completely static. I believe
the metadata itself can be static, and instead of an arbitrary build
script we can declare a build hook, but that’s an argument for a PEP!

I do believe a declarative build system can work for the 90% case
though and should probably be the “default” option.

It is indeed possible for most python packages, and I tried to figure out quite a few things to make that percentage as high as possible in Bento.

The really difficult part is not that, though, but making the system pluggable to allow pretty much arbitrary extensibility. What makes distutils very ill suited to that task is the whole subcommand model where options are not known in advance (that was the #1 issue when integrating distutils and scons).

For bento, I came up with a system of 'build manifest', where each command produces a text-based 'protocol' that other commands might use (https://github.com/pv/bento/blob/master/doc/source/hacking.rst). The implementation is not great but I think the idea is still sound: this means that in theory, you could build C extensions with something not involving python at all (e.g. make), but as long as the build process output that file, you could build installers from the system. IOW, the json file acts as 'middleware'.

What any future solution should *not* try to do is inventing yet another build system

David