Where is this flag coming from?
On my MacOS 10.2 system an unexpected flag is added to the link line for the bsddb module: gcc -bundle -bundle_loader python.exe \ build/temp.darwin-6.1-Power\ Macintosh-2.3/bsddbmodule.o \ -L/sw/lib -L/sw/lib -L/usr/local/lib -Wl,-R/sw/lib -ldb-4.1 \ -o build/lib.darwin-6.1-Power\ Macintosh-2.3/bsddb.so None of the other extension modules are linked with "-Wl,-R/sw/lib". I don't find "-R" in either setup.py or configure. Any idea where this is coming from and how to get rid of it? Skip
"SM" == Skip Montanaro <skip@pobox.com> writes:
SM> On my MacOS 10.2 system an unexpected flag is added to the SM> link line for the bsddb module: | gcc -bundle -bundle_loader python.exe \ | build/temp.darwin-6.1-Power\ Macintosh-2.3/bsddbmodule.o \ | -L/sw/lib -L/sw/lib -L/usr/local/lib -Wl,-R/sw/lib -ldb-4.1 \ | -o build/lib.darwin-6.1-Power\ Macintosh-2.3/bsddb.so SM> None of the other extension modules are linked with SM> "-Wl,-R/sw/lib". I don't find "-R" in either setup.py or SM> configure. Any idea where this is coming from and how to get SM> rid of it? Don't you remember, Skip!? I think we worked this out when we were looking at the linking problems with bsddb, since the from-source Berkeley install doesn't put its libs in a normally searched location. These flags get added because of the runtime_library_dirs arg in the Extension constructor for bsddbmodule.c. Are you saying that the flags are wrong, the paths are wrong, the flags are unnecessary, or something else? -Barry
SM> None of the other extension modules are linked with "-Wl,-R/sw/lib". SM> I don't find "-R" in either setup.py or configure. Any idea where SM> this is coming from and how to get rid of it? BAW> Don't you remember, Skip!? I think we worked this out when we were BAW> looking at the linking problems with bsddb, since the from-source BAW> Berkeley install doesn't put its libs in a normally searched BAW> location. I remember having problems with bsddb, which I think were fixed. -Wl,-R/sw/lib is clearly wrong on MacOS X though. Deleting it and running the link command manually works. BAW> Are you saying that the flags are wrong, the paths are wrong, the BAW> flags are unnecessary, or something else? The flags are incorrect: ... building 'bsddb' extension gcc -bundle -bundle_loader python.exe \ build/temp.darwin-6.1-Power\ Macintosh-2.3/bsddbmodule.o \ -L/sw/lib -L/home/skip/local/lib -L/sw/lib -L/usr/local/lib \ -Wl,-R/sw/lib -ldb-4.1 \ -o build/lib.darwin-6.1-Power\ Macintosh-2.3/bsddb.so ld: unknown flag: -R/sw/lib ... Skip
On Tue, 8 Oct 2002, Skip Montanaro wrote:
SM> None of the other extension modules are linked with "-Wl,-R/sw/lib". SM> I don't find "-R" in either setup.py or configure. Any idea where SM> this is coming from and how to get rid of it?
BAW> Don't you remember, Skip!? I think we worked this out when we were BAW> looking at the linking problems with bsddb, since the from-source BAW> Berkeley install doesn't put its libs in a normally searched BAW> location.
I remember having problems with bsddb, which I think were fixed. -Wl,-R/sw/lib is clearly wrong on MacOS X though. Deleting it and running the link command manually works.
BAW> Are you saying that the flags are wrong, the paths are wrong, the BAW> flags are unnecessary, or something else?
The -R flag in not supported on all systems. 2.2.2b1 build is broken on SCO Open Server now too.
The flags are incorrect:
... building 'bsddb' extension gcc -bundle -bundle_loader python.exe \ build/temp.darwin-6.1-Power\ Macintosh-2.3/bsddbmodule.o \ -L/sw/lib -L/home/skip/local/lib -L/sw/lib -L/usr/local/lib \ -Wl,-R/sw/lib -ldb-4.1 \ -o build/lib.darwin-6.1-Power\ Macintosh-2.3/bsddb.so ld: unknown flag: -R/sw/lib ...
Skip
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev
-- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net
TR> The -R flag in not supported on all systems. TR> 2.2.2b1 build is broken on SCO Open Server now too.
Shouldn't distutils be taught what to do with runtime_library_dirs for SCO and MacOSX?
My thoughts exactly. --Guido van Rossum (home page: http://www.python.org/~guido/)
>> Shouldn't distutils be taught what to do with runtime_library_dirs >> for SCO and MacOSX? Guido> My thoughts exactly. Given that MacOSX's linker doesn't seem to understand -R at all (no matter how you deliver it), I adjusted the code in runtime_library_dir_option (in unixccompiler.py) to return "-L" + dir instead of either of the "-R" possibilities. This is probably not as good as it could be, but I couldn't tease anything from the ld(1) man page which looked to me like an equivalent to "-R" or LD_RUN_PATH. Perhaps someone with more experience with MacOSX or link editors in general would like to take a peek at the problem. If so, let me know and I'll be happy to shoot you a copy of the ld(1) man page. Skip
On Wed, 9 Oct 2002, Barry A. Warsaw wrote:
"TR" == Tim Rice <tim@multitalents.net> writes:
TR> The -R flag in not supported on all systems. TR> 2.2.2b1 build is broken on SCO Open Server now too.
Shouldn't distutils be taught what to do with runtime_library_dirs for SCO and MacOSX?
It gets tricky trying to automagicly use the -R flag. As an example, the default install dir for OpenSSL is /usr/local/ssl. So on platforms that support -R you would think you could use -L/usr/local/ssl/lib -R/usr/local/ssl/lib -lssl -lcrypto OpenUNIX supports -R but if you did not build shared libraries for OpenSSL, the -R/usr/local/ssl/lib will break the build. Then there is the whole issue of what non system dirs to include. Maybe you have OpenSSL 0.9.6g in /usr/local/ssl but you're testing 0.9.7 that's installed in some other dir. If setup.py automagicly adds ssl to build the '_socket' extension, how does it know which version to use? How would it even find the one not in /usr/local/ssl? Normally you would configure --with-ssl-dir=/some_path_to_ssl As one who has done ports to many Open Source projects, I found python one of the more difficult ones because the build process uses python. So for a new platform (or broken build) you have to know python to build python.
-Barry
-- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net
"TR" == Tim Rice <tim@multitalents.net> writes:
TR> It gets tricky trying to automagicly use the -R flag. As an TR> example, the default install dir for OpenSSL is TR> /usr/local/ssl. So on platforms that support -R you would TR> think you could use -L/usr/local/ssl/lib -R/usr/local/ssl/lib TR> -lssl -lcrypto OpenUNIX supports -R but if you did not build TR> shared libraries for OpenSSL, the -R/usr/local/ssl/lib will TR> break the build. Since there seems to be so little agreement among *nixes about how to do this, it seems like the only/best place to codify all this knowledge is in distutils. I still think specifying runtime_library_dirs to the Extension constructor is the right thing to do. But maybe on some systems distutils ought to ignore it, or at least check to make sure that the dynamic libraries it wants to link against are actually present. TR> Then there is the whole issue of what non system dirs to TR> include. Maybe you have OpenSSL 0.9.6g in /usr/local/ssl but TR> you're testing 0.9.7 that's installed in some other dir. If TR> setup.py automagicly adds ssl to build the '_socket' TR> extension, how does it know which version to use? How would it TR> even find the one not in /usr/local/ssl? Normally you would TR> configure --with-ssl-dir=/some_path_to_ssl I would hope that setup.py would match the library version with the header version, but I'm sure it's not easy or foolproof. Look at the machinations setup.py goes through to build bsddbmodule. I still think it's best to do this so that Python has a chance of building out of the box for the widest possible audience. It's impossible to get 100% right. TR> As one who has done ports to many Open Source projects, I TR> found python one of the more difficult ones because the build TR> process uses python. So for a new platform (or broken build) TR> you have to know python to build python. Python's also a pretty complicated piece of software, what with each extension module linking against non-standard libraries presenting its own set of complications. But what tool that you can portably count on being present would you use instead? -Barry
barry@python.org (Barry A. Warsaw) writes:
Since there seems to be so little agreement among *nixes about how to do this, it seems like the only/best place to codify all this knowledge is in distutils. I still think specifying runtime_library_dirs to the Extension constructor is the right thing to do. But maybe on some systems distutils ought to ignore it, or at least check to make sure that the dynamic libraries it wants to link against are actually present.
I think both is difficult: It is hard to find out what option to use on what system, and how to find out whether a shared library is present. As a starting point, you have to decide whether you want a conservative strategy (white list of systems you know how to steer), or an optimistic strategy (black list of systems on which the standard strategy fails). For 2.2.2, backing out any setup.py changes to add runtime_library_dirs should be taking into consideration. Regards, Martin
[Martin]
For 2.2.2, backing out any setup.py changes to add runtime_library_dirs should be taking into consideration.
This sounds like a conservative solution. There is exactly one such option, for ssl. The checkin message: ---------------------------- revision 1.73.4.9 date: 2002/09/30 14:42:29; author: bwarsaw; state: Exp; lines: +2 -1 Add "runtime_library_dirs = ssl_libs" to the _socket Extension specification so that the proper runtime ld.so path gets compiled into the extension. This fixes _socket for Solaris systems with libssl and libcrypto in non-standard locations and should be fine for other systems as well. Closes SF bug #565710. Forward port candidate for Python 2.3 (I'll work on that). ---------------------------- The diff (apart from a change to __version__) is: *************** *** 364,369 **** --- 364,370 ---- exts.append( Extension('_socket', ['socketmodule.c'], include_dirs = ssl_incs, library_dirs = ssl_libs, + runtime_library_dirs = ssl_libs, libraries = ['ssl', 'crypto'], define_macros = [('USE_SSL',1)] ) ) else: Maybe we should be more conservative and add this option only for Solaris? How does one detect Solaris in setup.py? How about this patch (on top of the above one, not replacing it)? *** setup.py 7 Oct 2002 10:38:33 -0000 1.73.4.10 --- setup.py 9 Oct 2002 21:15:04 -0000 *************** *** 361,370 **** if (ssl_incs is not None and ssl_libs is not None): exts.append( Extension('_socket', ['socketmodule.c'], include_dirs = ssl_incs, library_dirs = ssl_libs, ! runtime_library_dirs = ssl_libs, libraries = ['ssl', 'crypto'], define_macros = [('USE_SSL',1)] ) ) else: --- 361,373 ---- if (ssl_incs is not None and ssl_libs is not None): + rtlibs = None + if platform.startswith('sunos'): + rtlibs = ssl_libs exts.append( Extension('_socket', ['socketmodule.c'], include_dirs = ssl_incs, library_dirs = ssl_libs, ! runtime_library_dirs = rtlibs, libraries = ['ssl', 'crypto'], define_macros = [('USE_SSL',1)] ) ) else: I hope that Skip/Barry can test this on MacOS X and Martin on Solaris! Note: the 2.3 setup.py contains other uses of runtime_library_dirs, which probably need to be treated the same way. Skip, can you cook up patches there? (The 2.2.2 fix for ssl also needs to be forward-ported!) --Guido van Rossum (home page: http://www.python.org/~guido/)
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> Maybe we should be more conservative and add this option only GvR> for Solaris? That was one suggestion I made when we talked about it last. Sounds like it's probably the right approach to whitelist the platforms we know are broken without it. I'm running a test on SF's compile farm Solaris machine now. GvR> How does one detect Solaris in setup.py? How about this GvR> patch (on top of the above one, not replacing it)? That's what I'm trying. GvR> I hope that Skip/Barry can test this on MacOS X and Martin on GvR> Solaris! I'll try OSX momentarily. GvR> Note: the 2.3 setup.py contains other uses of GvR> runtime_library_dirs, which probably need to be treated the GvR> same way. But maybe not. If you install any modern BerkeleyDB from source it always sticks it in a non-standard location. Hmm, maybe a blacklist is more useful for this one? Jeez this stuff sucks. GvR> Skip, can you cook up patches there? (The 2.2.2 GvR> fix for ssl also needs to be forward-ported!) Right, sorry, I'll do that if Guido's changes pans out for Solaris, Linux, and OSX. (Side note: I think I will have a few more minor patches for the email package, which I'll try to get into 2.2.2 tonight, if my cold will lay low for a few hours.) -Barry
"BAW" == Barry A Warsaw <barry@python.org> writes:
GvR> I hope that Skip/Barry can test this on MacOS X and Martin on GvR> Solaris! BAW> I'll try OSX momentarily. The patch seems fine on RH7.3, and Solaris 8. OSX build not yet done, but I suspect it'll work. If so, I'll commit the change to setup.py for 2.2.2 and forward port it to 2.3. -Barry
On Wed, 9 Oct 2002, Guido van Rossum wrote:
[Martin]
For 2.2.2, backing out any setup.py changes to add runtime_library_dirs should be taking into consideration.
This sounds like a conservative solution. There is exactly one such option, for ssl. The checkin message:
[snip]
+ runtime_library_dirs = ssl_libs,
The only real problem is -R seems to be added for all unix platforms. This change would benefit all my platforms if it handled -R properly In OpenSSH we only use -R on solaris and freebsd. Perhaps runtime_library_dir_option() in Lib/distutils/unixccompiler.py needs some work. -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net
"TR" == Tim Rice <tim@multitalents.net> writes:
TR> The only real problem is -R seems to be added for all unix TR> platforms. This change would benefit all my platforms if it TR> handled -R properly In OpenSSH we only use -R on solaris and TR> freebsd. Perhaps runtime_library_dir_option() in TR> Lib/distutils/unixccompiler.py needs some work. Anybody care to try a build on SF's freebsd system? If not, I'll try it after the Solaris build finishes. -Barry
On Wed, 9 Oct 2002, Barry A. Warsaw wrote:
"TR" == Tim Rice <tim@multitalents.net> writes:
TR> The only real problem is -R seems to be added for all unix TR> platforms. This change would benefit all my platforms if it TR> handled -R properly In OpenSSH we only use -R on solaris and TR> freebsd. Perhaps runtime_library_dir_option() in TR> Lib/distutils/unixccompiler.py needs some work.
I just tried Guido's setup.py patch on SCO Open Server and it works there. No more -R. :-) Am I out to lunch on the idea that the logic should be in Lib/distutils/unixccompiler.py where the -R is added?
Anybody care to try a build on SF's freebsd system? If not, I'll try it after the Solaris build finishes.
-Barry
-- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net
"TR" == Tim Rice <tim@multitalents.net> writes:
TR> I just tried Guido's setup.py patch on SCO Open Server TR> and it works there. No more -R. :-) Cool. I think we're squared away for Python 2.2.2. TR> Am I out to lunch on the idea that the logic should be in TR> Lib/distutils/unixccompiler.py where the -R is added? That's what Skip checked in but only for Python 2.3 it seems. I'm a little confused now about what should be ported from 2.2.2 to 2.3, but maybe I'm still foggied headed from my cold. The SF freebsd build of 2.2.2 seemed to work fine, except that test_sax failed for reasons I don't know and don't have the energy right now to investigate. 2.2.2 works for me on all of RH7.3, Solaris 8, and OSX 10.2 (except for test_locale on the latter which has already been brought up). Python 2.3cvs is working fine for me on RH7.3. The Solaris 8 and OSX builds are still running, and I can't do the freebsd build until the Solaris one ends (well, I could, but it's easier to just wait). -Barry
TR> Am I out to lunch on the idea that the logic should be in TR> Lib/distutils/unixccompiler.py where the -R is added? BAW> That's what Skip checked in but only for Python 2.3 it seems. I'm BAW> a little confused now about what should be ported from 2.2.2 to BAW> 2.3, but maybe I'm still foggied headed from my cold. I thought this was only a 2.3 thing. I just tried on the release22_maint CVS branch. The bsddb module compiled and linked fine, but I don't know why. Oh, wait a minute... The release22_maint branch doesn't have the bsddb bells and whistles I added to setup.py a couple months ago. Consequently, it's picking up the 1.85 version of the include file and library (which, I will remind people has known breakage). Should I backport those changes? I think what's in 2.3's setup.py is much more robust, but if people have been getting Berkeley DB 1.85 libraries in 2.2.1 and don't get a clean compile in 2.2.2 without editing setup.py because 1.85 is all they have available, that might be construed as "more than a bugfix". Let me know what you want to do in this regard Guido. Skip
I thought this was only a 2.3 thing.
But this thread started with you reporting a 2.2.2 problem trying to compile SSL!
I just tried on the release22_maint CVS branch. The bsddb module compiled and linked fine, but I don't know why. Oh, wait a minute... The release22_maint branch doesn't have the bsddb bells and whistles I added to setup.py a couple months ago. Consequently, it's picking up the 1.85 version of the include file and library (which, I will remind people has known breakage).
Should I backport those changes? I think what's in 2.3's setup.py is much more robust, but if people have been getting Berkeley DB 1.85 libraries in 2.2.1 and don't get a clean compile in 2.2.2 without editing setup.py because 1.85 is all they have available, that might be construed as "more than a bugfix".
Let me know what you want to do in this regard Guido.
I think it's too precarious to fix in 2.2.2. --Guido van Rossum (home page: http://www.python.org/~guido/)
>> I thought this was only a 2.3 thing. Guido> But this thread started with you reporting a 2.2.2 problem trying Guido> to compile SSL! Nope, it was bsddb on 2.3. Someone else was talking about SSL, which I could generally care less about. I use bsddb all the time, however. Skip
On Wed, 9 Oct 2002, Barry A. Warsaw wrote:
"TR" == Tim Rice <tim@multitalents.net> writes:
TR> I just tried Guido's setup.py patch on SCO Open Server TR> and it works there. No more -R. :-)
Cool. I think we're squared away for Python 2.2.2.
TR> Am I out to lunch on the idea that the logic should be in TR> Lib/distutils/unixccompiler.py where the -R is added?
That's what Skip checked in but only for Python 2.3 it seems.
The runtime_library_dir_option function seems to be identical in both 2.3 and 2.2.2
I'm a little confused now about what should be ported from 2.2.2 to 2.3, but maybe I'm still foggied headed from my cold.
[snip]
-Barry
-- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net
Tim> Am I out to lunch on the idea that the logic should be in Tim> Lib/distutils/unixccompiler.py where the -R is added? I don't think so. How would you adjust it to work for SCO? Skip
I just tried Guido's setup.py patch on SCO Open Server and it works there. No more -R. :-)
Am I out to lunch on the idea that the logic should be in Lib/distutils/unixccompiler.py where the -R is added?
Well, if we knew what to do on other systems instead of -R, that would help... I'm curious if perhaps Apple's loader (which started this) interprets -L as meaning both -L and -R in traditional Unix? --Guido van Rossum (home page: http://www.python.org/~guido/)
On Thursday, Oct 10, 2002, at 14:05 Europe/Amsterdam, Guido van Rossum wrote:
I'm curious if perhaps Apple's loader (which started this) interprets -L as meaning both -L and -R in traditional Unix?
For frameworks that is what happens, and while I haven't tried it for normal shared libraries the documentation suggest that this is indeed the case. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -
Guido> I'm curious if perhaps Apple's loader (which started this) Guido> interprets -L as meaning both -L and -R in traditional Unix? Damned if I could tell by reading the ld(1) manpage. The change I checked in to unixccompiler.py returns -L only because I figured "-L<dir>" would be innocuous, not because I expected it to have "-R<dir>" semantics. I'll ask around on the Pythonmac-SIG. Skip
Guido> I'm curious if perhaps Apple's loader (which started this) Guido> interprets -L as meaning both -L and -R in traditional Unix?
Damned if I could tell by reading the ld(1) manpage. The change I checked in to unixccompiler.py returns -L only because I figured "-L<dir>" would be innocuous, not because I expected it to have "-R<dir>" semantics.
Why did you check it in to Python 2.3 but not to 2.2.2? --Guido van Rossum (home page: http://www.python.org/~guido/)
"Guido" == Guido van Rossum <guido@python.org> writes:
Guido> I'm curious if perhaps Apple's loader (which started this) Guido> interprets -L as meaning both -L and -R in traditional Unix? >> >> Damned if I could tell by reading the ld(1) manpage. The change I >> checked in to unixccompiler.py returns -L only because I figured >> "-L<dir>" would be innocuous, not because I expected it to have >> "-R<dir>" semantics. Guido> Why did you check it in to Python 2.3 but not to 2.2.2? Because I couldn't build bsddb on 2.3 without it. Skip
Skip Montanaro <skip@pobox.com> writes:
Damned if I could tell by reading the ld(1) manpage. The change I checked in to unixccompiler.py returns -L only because I figured "-L<dir>" would be innocuous, not because I expected it to have "-R<dir>" semantics.
Thanks to 'otool -L', you'll see that the complete path of the library is embedded into the executable. Regards, Martin
participants (6)
-
barry＠python.org -
Guido van Rossum -
Jack Jansen -
martin＠v.loewis.de -
Skip Montanaro -
Tim Rice