[Numpy-discussion] [matplotlib-devel] Announcing toydist, improving distribution and packaging situation

[David Cournapeau]

On Sun, Jan 3, 2010 at 8:05 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Dec 29, 2009 at 6:34 AM, David Cournapeau <cournape at gmail.com> wrote:
>> Buildout, virtualenv all work by sandboxing from the system python:
>> each of them do not see each other, which may be useful for
>> development, but as a deployment solution to the casual user who may
>> not be familiar with python, it is useless. A scientist who installs
>> numpy, scipy, etc... to try things out want to have everything
>> available in one python interpreter, and does not want to jump to
>> different virtualenvs and whatnot to try different packages.
>
> What I do -- and documented for people in my lab to do -- is set up
> one virtualenv in my user account, and use it as my default python. (I
> 'activate' it from my login scripts.) The advantage of this is that
> easy_install (or pip) just works, without any hassle about permissions
> etc.

It just works if you happen to be able to build everything from
sources. That alone means you ignore the majority of users I intend to
target.

No other community (except maybe Ruby) push those isolated install
solutions as a general deployment solutions. If it were such a great
idea, other people would have picked up those solutions.

> This should be easier, but I think the basic approach is sound.
> "Integration with the package system" is useless; the advantage of
> distribution packages is that distributions can provide a single
> coherent system with consistent version numbers across all packages,
> etc., and the only way to "integrate" with that is to, well, get the
> packages into the distribution.

Another way is to provide our own repository for a few major
distributions, with automatically built packages. This is how most
open source providers work. Miguel de Icaza explains this well:

http://tirania.org/blog/archive/2007/Jan-26.html

I hope we will be able to reuse much of the opensuse build service
infrastructure.

>
> On another note, I hope toydist will provide a "source prepare" step,
> that allows arbitrary code to be run on the source tree. (For, e.g.,
> cython->C conversion, ad-hoc template languages, etc.) IME this is a
> very common pain point with distutils; there is just no good way to do
> it, and it has to be supported in the distribution utility in order to
> get everything right. In particular:
>  -- Generated files should never be written to the source tree
> itself, but only the build directory
>  -- Building from a source checkout should run the "source prepare"
> step automatically
>  -- Building a source distribution should also run the "source
> prepare" step, and stash the results in such a way that when later
> building the source distribution, this step can be skipped. This is a
> common requirement for user convenience, and necessary if you want to
> avoid arbitrary code execution during builds.

Build directories are hard to implement right. I don't think toydist
will support this directly. IMO, those advanced builds warrant a real
build tool - one main goal of toydist is to make integration with waf
or scons much easier. Both waf and scons have the concept of a build
directory, which should do everything you described.

David