[Python-Dev] Status of packaging in 3.3

[Dag Sverre Seljebotn]

On 06/22/2012 12:05 AM, Dag Sverre Seljebotn wrote:
> On 06/21/2012 11:04 PM, Tarek Ziadé wrote:
>> On 6/21/12 10:46 PM, Dag Sverre Seljebotn wrote:
>> ...
>>>> I think we should, as you proposed, list a few projects w/ compilation
>>>> needs -- from the simplest to the more complex, then see how a standard
>>>> *description* could be used by any tool
>>>
>>> It's not clear to me what you mean by description. Package metadata,
>>> install information or description of what/how to build?
>>>
>>> I hope you don't mean the latter, that would be insane...it would
>>> effectively amount to creating a build tool that's both more elegant
>>> and more powerful than any option that's currently already out there.
>>>
>>> Assuming you mean the former, that's what David did to create Bento.
>>> Reading and understanding Bento and the design decisions going into it
>>> would be a better use of time than redoing a discussion, and would at
>>> least be a very good starting point.
>>
>> What I mean is : what would it take to use Bento (or another tool) as
>> the compiler in a distutils-based project, without having to change the
>> distutils metadata.
>
> As for current distutils/setuptools/distribute metadata, the idea is you
> run the bento conversion utility to convert it to Bento metadata, then
> use Bento.
>
> Please read
>
> http://cournape.github.com/Bento/
>
> There may be packages where this doesn't work and you'd need to tweak
> the results yourself though.
>
>>> Here's the flip side: If you have zero knowledge about compilers, it's
>>> going to be almost impossible to have a meaningful discussion about a
>>> compilation PEP. It's very hard to discuss standards unless everybody
>>> involved have the necessary prerequisite knowledge. You don't go
>>> discussing details of the Linux kernel without some solid C experience
>>> either.
>> Consider me as the end user that want to have his 2 C modules compiled
>> in their Python project.
>
> OK, so can I propose that you kill off distutils2 and use bento
> wholesale instead?
>
> Obviously not. So you're not just an end-user. That illusion would wear
> rather thin very quickly.

I regret this comment, it's not helpful to the discussion. Trying again:

David's numscons project was a large effort and it tried to integrate a 
proper build system (scons) with distutils. That effort didn't in the 
end go anywhere.

But I think it did show that everything is coupled to everything, and 
that build system integration (and other "special" needs of the scipy 
community) affects everything in the package system.

It's definitely not as simple as having somebody with compiler 
experience chime in on the isolated topic of how to build extensions. 
It's something that needs to drive the entire design process. Which is 
perhaps why it is difficult to have a package system designed by people 
who don't know compilers to be usable by people who need to use them in 
non-trivial ways.

Dag

>
>>>
>>> The necessary prerequisites in this case is not merely "knowledge of
>>> compilers". To avoid repeating mistakes of the past, the prerequisites
>>> for a meaningful discussion is years of hard-worn experience building
>>> software in various languages, on different platforms, using different
>>> build tools.
>>>
>>> Look, these problems are really hard to deal with. Myself I have
>>> experience with building 2-3 languages using 2-3 build tools on 2
>>> platforms, and I consider myself a complete novice and usually decide
>>> to trust David's instincts over trying to make up an opinion of my own
>>> -- simply because I know he's got a lot more experience than I have.
>>>
>>> Theoretically it is possible to separate and isolate concerns so that
>>> one set of people discuss build integration and another set of people
>>> discuss installation. Problem is that all the problems tangle -- in
>>> particular when the starting point is distutils!
>>>
>>> That's why *sometimes*, not always, design by committee is the wrong
>>> approach, and one-man-shows is what brings technology forwards.
>>
>> I am not saying this should be designed by a commitee, but rather - if
>> such a tool can be made compatible with simple Distutils project, the
>> guy behind this tool can probably help on a PEP with feedback from a
>> larger audience than the sci community.
>>
>> What bugs me is to say that we live in two separate worlds and cannot
>> build common pieces. This is not True.
>
> I'm not saying it's *impossible* to build common pieces, I'm suggesting
> that it's not cost-effective in terms of man-hours going into it. And
> the problem isn't technical as much as social and the mix of people and
> skill sets involved.
>
> But David really made that decision for me when he left distutils-sig,
> I'm not going to spend my own time and energy trying to get decent
> builds shoehorned into distutils2 when he is busy working on a solution.
>
> (David already spent loads of time on trying to integrate scons with
> distutils (the numscons project) and maintained numpy.distutils and
> scipy builds for years; I trust his judgement above pretty much anybody
> else's.)
>
>>>> So, I reiterate my proposal, and it could also be expressed like this:
>>>>
>>>> 1/ David writes a PEP where he describes how Bento interact with a
>>>> project -- metadata, description files, etc
>>>> 2/ Someone from distutils2 completes the PEP by describing how
>>>> setup.cfg
>>>> works wrt Extensions
>>>> 3/ we see if we can have a common standard even if it's a subset of
>>>> bento capabilities
>>>
>>> bento isn't a build tool, it's a packaging tool, competing directly
>>> with distutils2. It can deal with simple distutils-like builds using a
>>> bundled build tool, and currently has integration with waf for
>>> complicated builds; integration with other build systems will
>>> presumably be added later as people need it (the main point is that
>>> bento is designed for it).
>> I am not interested in Bento-the-tool. I am interested in what such a
>> tool needs from a project to use it =>
>
> Again, you should read the elevator pitch at
> http://cournape.github.com/Bento/ + the Bento documentation.
>
>> "It can deal with simple distutils-like builds using a bundled build
>> tool" => If I understand this correctly, does that mean that Bento can
>> build a distutils project with the distutils Metadata ?
>
> Sorry, what I meant with "distutils-like builds" is "two simple C
> extensions", i.e. the trivial build case.
>
> Dag