[Distutils] Installing Numeric and linking against libpython.so

[Greg Ward]

[Jim -- this discussion deserves to be on the SIG, so I'm cc'ing it
there; SIG readers -- if it looks like you missed a message you did:
Jim replied straight to me, but I'm putting this thread back on the
SIG.]

[Jim Preston explains the problem in the first place]
> I am trying to upgrade my Numeric, etc. and I seem to have a problem
> with the distutils
> setup.  Everything appears to install fine, but when I import Numeric, I
> get the message
> >> import Numeric
> Traceback (innermost last):
>   File "<stdin>", line 1, in ?
>   File "/usr/local/lib/python1.5/site-packages/Numeric/Numeric.py", line
> 76, in ?
>     import multiarray
> ImportError: ld.so.1: python: fatal: relocation error: file
> /usr/local/lib/python1.5/site-packages/Numeric/multiarray.so: symbol
> PySequence_Length: referenced symbol not found
> >>>

[me, bugging Jim for more info]
> What version of Python is this?  (And, for that matter, which version of
> NumPy and Distutils?  I'm assuming 15.2 and 0.1.3, but you never
> know...)

[Jim coughs up the details]
> Python 1.5.2 (latest, AFAIK), Numeric 15.2 and distutils 0.1.3.

[me again]
> How was your libpython built -- is it static or dynamic?  (Look in
> $exec_prefix/lib/python1.5/config to see -- you're looking for either
> libpython.a or libpython.so.)

[Jim]
> RIght. The default configuration was static, I tried to create libpython.so
> to try and get the linkage going, but no luck there either.

[me]
> Are you able to import any of the other NumPy extensions?  Have you run
> "ldd" on them to see if they were linked against libpython.so (assuming
> libpython.so is what you have)?  Have you run "nm" or "elfdump" to see
> if you can figure out what symbols from libpython are expected by the
> extensions?

[Jim]
> ldd tells me nothing, and nm tells me that all of the Py_Sequence functions,
> etc. are undefined.

[me]
> Did you try hacking (just cut and paste it) the link command used by
> Distutils to create multiarray.so to add
> "-L$exec_prefix/lib/python1.5/config -lpython" (where $exec_prefix is
> Python's prefix, which of course you'll have to interpolate manually),
> and see if that helps?

[Jim]
> No change.

OK, I'm stumped.  For those of you who missed his first post, Jim is
running Solaris 2.6, which just happens to be what I have on my machine
it work.  So I can swear up and down that NumPy 15.2 builds just fine
with Distutils 0.1.3 under Python 1.5.2 and Solaris 2.6.  Something
seems to be wrong with multiarray.so (one of the extensions installed by 
NumPy into site-packages/Numeric) in Jim's installation, but for me it
works great:

  $ python
  Python 1.5.2 (#3, Feb 11 2000, 11:14:47)  [GCC 2.8.1] on sunos5
  Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
  >>> import multiarray
  >>> dir(multiarray)
  ['__doc__', '__file__', '__name__', '__version__', 'argmax', 'argsort', 'array', 'arraytype', 'binarysearch', 'choose', 'concatenate', 'cross_correlate', 'error', 'fromstring', 'innerproduct', 'repeat', 'reshape', 'set_string_function', 'sort', 'take', 'transpose', 'zeros']

See, It Just Works (TM).  IOW, when my Python loads multiarray.so, it
has no problem finding symbols like 'PySequence_Length'.  My copy of
multiarray.so can't possibly have all of Python linked into it -- it's
only 70k.  Anyways, that would be so colossally stupid it doesn't even
bear thinking about.  "ldd" on multiarray.so reveals that it depends on
the usual Solaris shared libs -- libc.so, libelf.so, libucb.so,
libsocket.so, and a few others.  All of them are in /usr/lib, so clearly
nothing Python-specific there.  

But Jim gives one last hint:
> I keep thinking that there is some old stuff hanging around from earlier
> python releases that is messing this up, but I have tried quite hard to get
> rid of everything and build a clean installation and I am still having the same
> problems.

OK, have you done this, with a *fresh*, brand-new Python 1.5.2 source
distribution:

  $ ./configure --prefix=/tmp/usr
  $ [edit the setup file, you know the routine]
  $ make
  $ make install

and then unpack a fresh, brand-new Distutils 0.1.3 distribution and
install it to the temporary Python installation

  $ /tmp/usr/bin/python setup.py install    # in the Distutils-0.1.3 directory

and then unpack a fresh, brand-new NumPy 15.2 distribution ...

  $ /tmp/usr/bin/python setup.py install    # in the Numerical-15.2 directory

I hate having to say "reinstall Python is the answer", but I don't see
what could be going wrong except for a screwed up Python installation.
Please try this and let us know if it works or not -- either way, I want
to know!

[Jim asks a relevant question]
> I am not a shared library guru, by any means. How is it supposed to
> work, anyway? Is it the case that the multiarraymodule should be able
> to call functions from libpython,a without any such linkage specified,
> simply because it is being imported into a program (python) that is
> already linked against libpython.a ?  I must say that I am quite
> baffled here.

That's a good question, and I'm not entirely sure how it works.  My
gut-level intuition says that because python is the binary that's
loading multiarray.so into memory, all of Python's API is already there
-- so it Just Works.  (At least, for *me* it just works.)  IOW, if you
tried to load multiarray.so into Tcl or Perl or another C program, then
of course it wouldn't be able to resolve PySequence_Length.

But I would like to hear from someone who really understands this what's
going on.  I'm curious too!

> I am using gcc-2.95.1, with gnu ld, etc.

Hmmm, there's one difference between Jim's setup and mine -- we're still
using gcc 2.8.1, and the linker used to create shared objects is the
system linker -- /usr/ucb/ld.  This comes from Python's Makefile,
ie. this is how Python was built, so that's how Distutils builds
extensions.  If you build Python with a different linker, Distutils will 
build extensions with that linker.  This might be key!

        Greg
-- 
Greg Ward - all-purpose geek                            gward@python.net
http://starship.python.net/~gward/
God is real, unless declared integer.