[Python-Dev] Status of packaging in 3.3

[Dag Sverre Seljebotn]

On 06/21/2012 03:23 PM, Tarek Ziadé wrote:
> On 6/21/12 2:45 PM, Dag Sverre Seljebotn wrote:
>>
>> Guido was asked about build issues and scientific software at PyData
>> this spring, and his take was that "if scientific users have concerns
>> that are that special, perhaps you just need to go and do your own
>> thing". Which is what David is doing.
>>
>> Trailing Q&A session here: http://www.youtube.com/watch?v=QjXJLVINsSA
>
> if you know what you want and have a tool that does it, why bother using
> distutils ?
>
> But then, what your community will do with the guy that create packages
> with distutils ? just tell him he suck ?
>
> The whole idea is *interoperability*, not the tool used.
>
>>
>> Generalizing a bit I think it's "web developers" and "scientists"
>> typically completely failing to see each others' usecases. I don't
>> know if that bridge can be crossed through mailing list discussion
>> alone. I know that David tried but came to a point where he just had
>> to unsubscribe to distutils-sig.
> I was there, and sorry to be blunt, but he came to tell us we had to
> drop distutils because it sucked, and left because we did not follow
> that path
>
>
>>
>> Sometimes design by committee is just what you want, and sometimes
>> design by committee doesn't work. ZeroMQ, for instance, is a great
>> piece of software resulting from dropping out of the AQMP committee.
>>
>>>
>>> That will not work. And I will say here again what I think we should do
>>> imho:
>>>
>>> 1/ take all the packaging PEPs and rework them until everyone is happy
>>> (compilation sucks in distutils ? write a PEP !!!)
>>
>> I think the only way of making scientists happy is to make the build
>> tool choice arbitrary (and allow the use of waf, scons, cmake, jam,
>> ant, etc. for the build). After all, many projects contains more C++
>> and Fortran code than Python code. (Of course, one could make a PEP
>> saying that.)
>>
>> Right now things are so horribly broken for the scientific community
>> that I'm not sure if one *can* sanely specify PEPs. It's more a
>> question of playing around and throwing things at the wall and see
>> what sticks -- 5 years from now one is perhaps in a position where the
>> problem is really understood and one can write PEPs.
>>
>> Perhaps the "web developers" are at the PEP-ing stage already. Great
>> for you. But the usecases are really different.
> If you sit down and ask your self: "what are the information a python
> project should give me so I can compile its extensions ?" I think this
> has nothing to do with the tools/implementations.

I'm not sure if I understand. A project can't "give the information 
needed to build it". The build system is an integrated piece of the code 
and package itself. Making the build of library X work on some ugly HPC 
setup Y is part of the development of X.

To my mind a solution looks something like (and Bento is close to this):

  Step 1) "Some standard" to do configuration of a package (--prefix and 
other what-goes-where options, what libraries to link with, what 
compilers to use...)

  Step 2) Launch the package's custom build system (may be Unix shell 
script or makefile in some cases (sometimes portability is not a goal), 
may be a waf build)

  Step 3) "Some standard" to be able to cleanly 
install/uninstall/upgrade the product of step 2)

An attempt to do Step 2) in a major way in the packaging framework 
itself, and have the package just "declare" its C extensions, would not 
work. It's fine to have a way in the packaging framework that works for 
trivial cases, but it's impossible to create something that works for 
every case.

>
> And if we're able to write down in a PEP this, e.g. the information a
> compiler is looking for to do its job, then any tool out there waf,
> scons, cmake, jam, ant, etc, can do the job, no ?
>
>
>>
>> Anyway: I really don't want to start a flame-war here. So let's accept
>> up front that we likely won't agree here; I just wanted to clarify my
>> position.
> After 4 years I still don't understand what "we won't agree" means in
> this context. *NO ONE* ever ever came and told me : here's what I want a
> Python project to describe for its extensions.

That's unfortunate. To be honest, it's probably partly because it's 
easier to say what won't work than come with a constructive suggestion. 
A lot of people (me included) just use waf/cmake/autotools, and forget 
about making the code installable through PyPI or any of the standard 
Python tools. Just because that works *now* for us, but we don't have 
any good ideas for how to make this into something that works on a wider 
scale.

I think David is one of the few who has really dug into the matter and 
tried to find something that can both do builds and work through 
standard install mechanisms. I can't answer for why you haven't been 
able to understand one another.

It may also be an issue with how much one can constructively do on 
mailing lists. Perhaps the only route forward is to to bring people 
together in person and walk distutils2 people through some hairy 
scientific HPC builds (and vice versa).

> Just "we won't agree" or "distutils sucks" :)
>
>
> Gosh I hope we will overcome this lock one day, and move forward :D

Well, me too.

Dag