[Numpy-discussion] include file location and .eggs

[Andrew Straw]

konrad.hinsen at laposte.net wrote:

> On Jan 23, 2006, at 18:07, Andrew Straw wrote:
>
>> Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5.
>> You'd make sure the call to setup() (which is now imported from
>> setuptools, not distutils.core) included the following:
>
>
> ...
>
> OK, that looks sophisticated enough to me :-)
>
> Since setuptools keeps track of dependencies, it should be able to 
> provide on demand a list of all include file directories for all 
> packages on which the current build procedure depends, directly or 
> indirectly. To stay within my example, a package that depends on 
> ScientificPython, which in turn depends on Numeric, should be able to 
> get, with a single call, the include directories for *both* 
> ScientificPython and Numeric.
>
> Assuming such a mechanism exists, and is also implemented in plain 
> distutils (returning just the standard installation paths), then 
> personally I wouldn't care where header files are, because there is a 
> straightforward way to obtain that information.

The problem is backwards compatibility. Nothing like this is implemented
in plain distutils, which is why the packages will have to support it
themselves. This way they can keep working in both old distutils-based
environments and new ones, including with setuptools. The current
solution _works_ on yesteryear's Python 2.3. In my opinion, relying on
distutils to manage header file locations was a mistake, but that's easy
to say in hindsight, when it's clear that, as useful as distutils is, it
has some deficiencies.

Distutils never handled the case when, for example you installed Numeric
with '/usr/local/bin/python2.3 setup.py install
--prefix=/home/astraw/py23'. In this case, sure, the package got
installed where I wanted (in my $PYTHONPATH) in
/home/astraw/py23/lib/python2.3/site-packages/Numeric, but the headers
were in /home/astraw/py23/include/python2.3/Numeric. Distutils would
never find those headers again. This is exactly the same problem, and
it's one that distutils never solved, and I don't think we should hold
our breath waiting for Python 2.5's distutils to solve it. For one
thing, it's already solved at the package level a la numpy. For another,
given that it's already solved, who's going to bother attempting to make
a patch (which can't be relied on by older pythons) for Python 2.5,
which may not even be accepted?

>
> However, that is not what is currently being discussed for numpy, 
> unless I missed something. What is being discussed is an arrangement 
> that requires the setup.py of every package to reconstruct the 
> dependency tree and figure out where each package in the dependency 
> tree keeps its header files.

Why is "import numpy; include_dirs=numpy.get_numpy_include()" in
setup.py unpalatable for you? That's all that is required in setup.py.
To me, this is a small price to pay for the benefits gained. If you want
to argue this is more pain than you're willing to deal with, I think
that, to be fair, you need to give a counter-example whereby you explain
how you handle all those versions of Numeric and Scientific installed
simultaneously and how you select among them. I'll admit that in the
case you gave, of  2 versions of Scientific and 2 versions of Numeric
compiled into all 4 possible combinations, you'd also need an "import
pkg_resources; pkg_resources.require('Numeric==x.y')" prior to the
"import Numeric". I see the case you gave as a rather extreme case --
normally I would like, for example, a single version of (old) scipy
compiled against old Numeric and a single new version of scipy compiled
against numpy. (If it's not already working, I'm almost willing to
backport egg support to old scipy to do just this...)

Like I said above, distutils has no such equivalent mechanism, and this
_allows_ something that simply wasn't possible before and is arguably
broken in distutils.

>
>
> There is something else that I don't understand. If egg installation 
> uses setup from setuptools instead of distutils, it should be trivial 
> for the setup function from setuptools to implement install_headers 
> in a way that suits its need. So why request that numpy (and other 
> packages) adapt to setuptools, creating compatibility issues with 
> standard distutils, rather than modify setuptools and have it work 
> automatically for all packages?

That's an interesting idea. The short answer is that setuptools doesn't
do it currently. Perhaps it could. I'm not going to implement it,
though. It also doesn't solve the problem I gave above with plain
distutils, which I think is a real problem that numpy solves in a
backwards compatible way.