[Numpy-discussion] Announcing toydist, improving distribution and packaging situation

[René Dudfield]

On Tue, Dec 29, 2009 at 11:20 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Wed, Dec 30, 2009 at 3:36 AM, René Dudfield <renesd at gmail.com> wrote:
>> On Tue, Dec 29, 2009 at 2:34 PM, David Cournapeau <cournape at gmail.com> wrote:
>>> On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield <renesd at gmail.com> wrote:
>>>
>>>> Buildout is what a lot of the python community are using now.
>>>
>>> I would like to note that buildout is a solution to a problem that I
>>> don't care to solve. This issue is particularly difficult to explain
>>> to people accustomed with buildout in my experience - I have not found
>>> a way to explain it very well yet.
>>
>> Hello,
>>
>> The main problem buildout solves is getting developers up to speed
>> very quickly on a project.  They should be able to call one command
>> and get dozens of packages, and everything else needed ready to go,
>> completely isolated from the rest of the system.
>>
>> If a project does not want to upgrade to the latest versions of
>> packages, they do not have to.  This reduces the dependency problem a
>> lot.  As one package does not have to block on waiting for 20 other
>> packages.  It makes iterating packages daily, or even hourly to not be
>> a problem - even with dozens of different packages used.  This is not
>> theoretical, many projects iterate this quickly, and do not have
>> problems.
>>
>> Backwards compatibility is of course a great thing to keep up... but
>> harder to do with dozens of packages, some of which are third party
>> ones.  For example, some people are running pygame applications
>> written 8 years ago that are still running today on the latest
>> versions of pygame.  I don't think people in the python world
>> understand API, and ABI compatibility as much as those in the C world.
>>
>> However buildout is a solution to their problem, and allows them to
>> iterate quickly with many participants, on many different projects.
>> Many of these people work on maybe 20-100 different projects at once,
>> and some machines may be running that many applications at once too.
>> So using the system pythons packages is completely out of the question
>> for them.
>
> This is all great, but I don't care about solving this issue, this is
> a *developer* issue. I don't mean this is not an important issue, it
> is just totally out of scope.
>
> The developer issues I care about are much more fine-grained (corrent
> dependency handling between target, toolchain customization, etc...).
> Note however that hopefully, by simplifying the packaging tools, the
> problems you see with numpy on 2.6 would be less common. The whole
> distutils/setuptools/distribute stack is hopelessly intractable, given
> how messy the code is.
>

The numpy issue is because of the change in package handling for 2.6,
which numpy 1.3 was not developed for.



>>
>> It is very easy to include a dozen packages in a buildout, so that you
>> have all the packages required.
>
> I think there is a confusion - I mostly care about *end users*. People
> who may not have compilers, who want to be able to easily upgrade one
> package, etc...
>

I was just describing the problems that buildout solves(for others).
If I have a project that depends on numpy and 12 other packages, I can
send it to other people who can get their project up and running
fairly quickly (assuming everything installs ok).

btw, numpy 1.4 works with buildout! (at least on my ubuntu box)  sweet :)

cd /tmp/
bzr branch lp:numpybuildout
cd numpybuildout/trunk/

python bootstrap.py -d
./bin/buildout
./bin/py

>>> import numpy
>>> numpy.__file__
'/tmp/numpybuildout/trunk/eggs/numpy-1.4.0-py2.6-linux-i686.egg/numpy/__init__.pyc'