[Numpy-discussion] Announcing toydist, improving distribution and packaging situation

[Dag Sverre Seljebotn]

David wrote:
> Repository
> ========
>
> The goal here is to have something like CRAN
> (http://cran.r-project.org/web/views/), ideally with a build farm so
> that whenever anyone submits a package to our repository, it would
> automatically be checked, and built for windows/mac os x and maybe a
> few major linux distributions. One could investigate the build service
> from open suse to that end (http://en.opensuse.org/Build_Service),
> which is based on xen VM to build installers in a reproducible way.

Do you here mean automatic generation of Ubuntu debs, Debian debs, Windows
MSI installer, Windows EXE installer, and so on? (If so then great!)

If this is the goal, I wonder if one looks outside of Python-land one
might find something that already does this -- there's a lot of different
package format, "Linux meta-distributions", "install everywhere packages"
and so on.

Of course, toydist could have such any such tool as a backend/in a pipeline.

> What's next ?
> ==========
>
> At this point, I would like to ask for help and comments, in particular:
>  - Does all this make sense, or hopelessly intractable ?
>  - Besides the points I have mentioned, what else do you think is needed ?

Hmm. What I miss is the discussion of other native libraries which the
Python libraries need to bundle. Is it assumed that one want to continue
linking C and Fortran code directly into Python .so modules, like the
scipy library currently does?

Let me take CHOLMOD (sparse Cholesky) as an example.

 - The Python package cvxopt use it, simply by linking about 20 C files
directly into the Python-loadable module (.so) which goes into the Python
site-packages (or wherever). This makes sure it just works. But, it
doesn't feel like the right way at all.

 - scikits.sparse.cholmod OTOH simple specifies libraries=["cholmod"], and
leave it up to the end-user to make sure it is installed. Linux users
with root access can simply apt-get, but it is a pain for everybody else
(Windows, Mac, non-root Linux).

 - Currently I'm making a Sage SPKG for CHOLMOD. This essentially gets the
job done by not bothering about the problem, not even using the
OS-installed Python.

Something that would spit out both Sage SPKGs, Ubuntu debs, Windows
installers, both with Python code and C/Fortran code or a mix (and put
both in the place preferred by the system in question), seems ideal. Of
course one would still need to make sure that the code builds properly
everywhere, but just solving the distribution part of this would be a huge
step ahead.

What I'm saying is that this is a software distribution problem in
general, and I'm afraid that Python-specific solutions are too narrow.

Dag Sverre