[ python-Bugs-832799 ] Please link modules with shared lib

[SourceForge.net]

Bugs item #832799, was opened at 2003-10-30 02:36
Message generated for change (Settings changed) made by loewis
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=832799&group_id=5470

Category: Build
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: benson margulies (benson_basis)
>Assigned to: Martin v. Löwis (loewis)
Summary: Please link modules with shared lib

Initial Comment:
We've been working with libraries that are loaded with 
dlopen (on Linux and Solaris, the land of ELF) and 
which, in turn, use the embedded python interpreter.

Due to the behavior of the dynamic linker, this would 
work much better if modules were, by default, linked 
with -lpython2.3 instead of just left with hanging 
undefined symbols. Here's why.

The main executable isn't linked with python, and none 
of it's direct dependents are. So, the python symbols 
aren't in the global namespace.

The dlopened library is linked with python. However, 
when the dlopened library dlopens the modules, the 
linux linker is not clever enough to allow the second-
order library to use symbols from its parent. (Solaris 
has such a feature, but not linux). So, one has to 
manually dlopen the python library with RTLD_GLOBAL 
to make it work.

If each module had a NEED for the python lib (via -l at 
linktime), all this would just work.

I've got some local patches to build_ext.py for this 
purpose, but it would be nice to have official support.


----------------------------------------------------------------------

Comment By: benson margulies (benson_basis)
Date: 2003-11-25 22:53

Message:
Logged In: YES 
user_id=876734

I only see one possible issue with the patch.

I exploited the existing BLDLIBRARY macro. It says '-L.' on 
linux. Which is fine for the initial 'make', but not too great 
when the Makefile gets copied to the install dir.

The comments in the config dir leave me thinking that there 
would be a Makefile.pre in there, but there isn't. Just a 
Makefile.

The RUNSHARED macro ends up set back to where I built it, 
not to where I installed it. If some process would fix 
RUNSHARED, it could also fix BLDSHARED.


----------------------------------------------------------------------

Comment By: benson margulies (benson_basis)
Date: 2003-11-25 22:27

Message:
Logged In: YES 
user_id=876734

I submitted a set of patches that work for the initial build. I 
think I'll need more for the tools that are used later, I'm 
moving on to that next.


----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-11-24 23:16

Message:
Logged In: YES 
user_id=21627

1) Basically, you lose. If you don't want to build a module
as a shared library, you can build it statically, through
Modules/Setup. If you absolutely don't want to build a
module at all, you edit setup.py, and modify
disabled_module_list. If you don't want to edit
disabled_module_list, you build the module, and delete it
when done.

2) Using /usr/local/lib could be replaced, but I would
consider this out of scope of this change. Feel free to
submit a separate patch. Make sure that the alternative
patch manages to pick up shared libraries in all cases where
it finds them today.

3) 'cvs annotate' reveals that this was added in setup.py
1.100. 'cvs log' reveals that this was added in response to
python.org/sf/589427, where the Debian maintainer complains
that /usr/include is added to the list of -I options, even
though the compiler already has /usr/include in its search list.

----------------------------------------------------------------------

Comment By: benson margulies (benson_basis)
Date: 2003-11-23 21:44

Message:
Logged In: YES 
user_id=876734

I'm running into issues trying to come up with a clean version 
of this. I'd like to know what you think of some of these 
before I try to port what I've got into 2.4 and come up with 
patches.

1) setup.py seems to have no way to be selective about 
which modules to build. What if I don't want to try (and then 
fail make install) to build, for example, ssl?

2) setup.py makes assumptions about pathnames. It always 
puts /usr/local/lib into the build path. On a 64-bit solaris or HP 
system, this can lead to a mess, if the 64 bit libraries are 
somewhere else.

3) There is an existing provision to add additional libs to the 
build in setup.py, but it's disabled if the prefix is /usr. Why?

I have this feeling that someone had a plan here that I'm too 
obtuse to make out which would offer a shorter path to a 
solution then I'm getting tangled up in, for managing the build 
of these modules. 


----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-10-31 14:42

Message:
Logged In: YES 
user_id=21627

You can probably rely on libpythonxy.so ending up in
$(DESTDIR)$(LIBDIR)/$(INSTSONAME), whose values you can
retrieve from the installed Makefile (i.e. through
distutils.config).

----------------------------------------------------------------------

Comment By: benson margulies (benson_basis)
Date: 2003-10-31 13:22

Message:
Logged In: YES 
user_id=876734

To turn my patches into things that anyone else would want, 
I need to know what to put in the -L that will be required. At 
the time we're building the main python distro, that's 
straightforward. When a user builds an individual module, is 
there a sensible path relative to PYTHONHOME I can use?

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-10-31 11:05

Message:
Logged In: YES 
user_id=21627

Please do provide patches. Take into account that modules
may be built either with distutils, or through Modules/Setup.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=832799&group_id=5470