On Mar 27, 2014, at 4:19 PM, Paul Moore wrote:

On 27 March 2014 19:48, Donald Stufft wrote:

...

I do believe a declarative build system can work for the 90% case though and should probably be the "default" option.

+1. Easy things should be easy, and hard things possible. Ultimately, I want to say "this is my project, here are the Python files. And oh, here's a couple of C files, nothing clever here, move along.

When I interface with an external library that may not be in the same place on every system, I'd expect to say a bit more, but mainly just "look in location X, but the user can override this with a command line argument”.

That’s the job of the linker, it has built in locations to look for libraries and env vars/command line flags to override that. Someone packaging a C-ext that needs a library to compile shouldn’t be specifying where to find it anyways.

Going beyond that - autodetecting library locations, languages other than C (for example Cython), detecting CPU capabilities, etc - I would expect to be able to do it if I needed to, but I would not expect the declarative framework to be catering for those cases, I'd assume I needed to write code.

Paul

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Thu, Mar 27, 2014 at 7:48 PM, Donald Stufft wrote:

On Mar 27, 2014, at 3:42 PM, Daniel Holth wrote:

...

I became convinced that build was an inherently arbitrary-code process, and not something to be universally handled by a declarative system, when I observed an autotools project under configuration. The things spend ten minutes writing and compiling snippets of C code to determine which features are and are not supported by the runtime. This is *very* arbitrary code and probably one of the tamer things people like to do during builds.

As usual I blame distutils, because the separation is not there. So when you blame setup.py for having to run procedural code to generate the *metadata* you might go to far and think you should also eliminate a procedural build - simply because build and metadata are not adequately separate in the distutils design. Declarative build systems are a nice idea but they are not going to work for everyone. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Well for the record auto tools does all of that because there’s near zero ability to introspect the compiler to figure out what it supports except by compiling little programs that depend on X feature and then running them to see if they work or not.

That being said, yes building code is inherently a process that needs to execute things and cannot be made completely static. I believe the metadata itself can be static, and instead of an arbitrary build script we can declare a build hook, but that’s an argument for a PEP!

I do believe a declarative build system can work for the 90% case though and should probably be the “default” option.

It is indeed possible for most python packages, and I tried to figure out quite a few things to make that percentage as high as possible in Bento. The really difficult part is not that, though, but making the system pluggable to allow pretty much arbitrary extensibility. What makes distutils very ill suited to that task is the whole subcommand model where options are not known in advance (that was the #1 issue when integrating distutils and scons). For bento, I came up with a system of 'build manifest', where each command produces a text-based 'protocol' that other commands might use ( https://github.com/pv/bento/blob/master/doc/source/hacking.rst). The implementation is not great but I think the idea is still sound: this means that in theory, you could build C extensions with something not involving python at all (e.g. make), but as long as the build process output that file, you could build installers from the system. IOW, the json file acts as 'middleware'. What any future solution should *not* try to do is inventing yet another build system David

On Thu, Mar 27, 2014 at 2:23 PM, Nick Coghlan wrote:

On 28 Mar 2014 05:42, "Daniel Holth" wrote:

...

I became convinced that build was an inherently arbitrary-code process, and not something to be universally handled by a declarative system,

It wasn't an accident that last years PyCon panel was subtitled "setup.py *install* must die" :)

As others have suggested, declarative build will never be more than an 80% solution, and then beyond that, it should be a question of plumbing to invoke the project's own build system (and ideally that plumbing will be per build system, not per project).

Agreed -- I have been poking at this a bit, and trying to make gattai work for me: http://sourceforge.net/projects/gattai/ (It's a by and for python build system that essentially invokes the native build systems : make, setup.py, nmake, etc... But that's not really the point). the point is that I hit a wall pretty quickly with its declarative approach (JSON files). I find I can't do what I need to do with straight declaration (and I'm hacking gattai to support that) However, for the most part, all I need to be able to do is run arbitrary code to set up the declarations. The stuff that's been talked about: finding the right libraries to link to, that sort of thing. I actually think that aspect of the setup.py approach works pretty well, even though it does sometimes get kind of messy. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov