setup.py build_ext --rpath behaviour

Hello Distutils allows you to use a handy --rpath option to the build_ext command to add an RPATH value into the linked object. However some of the semantics are not great IMHO. For the first issue some background first. On SysV-like systems (systems using the ELF binary format) the RPATH field was created in the .dynamic section to allow the shared object to specify extra locations to look for their required shared objects. However this did overwrite the use of the LD_LIBRARY_PATH and that was no good for administrators, so another new field was added: RUNPATH. This was only used *after* LD_LIBRARY_PATH instead of before it solving the problem. When both fields are present the runtime linker ignores RPATH (it is impossible to create a shared object with a RUNPATH but no RPATH). The Solaris linker decided to always encode RUNPATH into the shared object whenever an RPATH is specified (using the -R option). This is a very sensible option most likely the correct behaviour. The GNU linker however decided that backwards compatibility was more important and to make it add the RUNPATH field you have to pass --enable-new-dtags to the linker, if you use -R or -rpath on it's own you will only get an RPATH, no RUNPATH. Now when using the --rpath option to the build_ext command distutils.unixcompiler.UnixCCompiler.runtime_library_dir_option() will use some heuristics to figure out which option to pass to compiler to get the runpath in (i.e. "-R" or "-Wl,-R" etc). I'm going to argue that this needs to be extened pass in -Wl,--enable-new-dtags,-R if the GNU linker is used so that the newer and better RUNPATH gets put into the shared objects all the time. (I don't yet know how to detect the GNU linker but would like a consensus on the desired behaviour before looking into this). Note that this is not completely a disaster, thanks to distutils.sysconfig.customize_compiler() adding the contents of the LDFLAGS environment variable to the command line of the linker invocation you can work around this currently. The second issue with build_ext --rpath is on AIX. Again some background on AIX shared objects, AIX is not SysV-like and uses the XCOFF binary format instead of ELF. Therefore they don't have a RPATH or RUNPATH, but they do have a think called LIBPATH which does something similar. The difference between XCOFF's LIBPATH and ELF's RUNPATH is that AIX's runtime linker does not have a default search path, hence the full search path needs to be encoded into the LIBPATH of the shared object. Now I would propose for build_ext --rpath to encode the LIBPATH when used on AIX since that is the correct thing to do IMHO. (Implementation note: since this is done by passing -blibpath:/opt/example.com/lib:/usr/lib:/lib to the linker note that distutils.unixccompiler.UnixCCompiler.link() would have to be changed not to strip out /usr/lib and /lib from the runtime_library_dirs on AIX. And distutils.unixccompiler.UnixCCompiler.runtime_library_dir_option() would have to use -blibpath:... or -Wl,-blibpath as appropriate) Again, this can currently be circumvented by using the LDFLAGS environment variable. Do these improvements sound sensible? And if so should I create one patch for each (and two bug reports) or combine them into one patch? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Thu, Apr 16, 2009 at 12:42 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
Do these improvements sound sensible? And if so should I create one patch for each (and two bug reports) or combine them into one patch?
They sound reasonable, please fill two differents bugs with your patches and if you can, add some unit tests. As long as you provide some support to test the final vesion on the various platforms (eg being a beta tester of the distutils trunk), I am able to work on your patches, and on the tests. I have not been very active on this command for the moment because of my need of beta-testers for the various platforms you mentioned. I do have a vmware running for Windows, Linux and Mac but that 's it so far Regards Tarek -- Tarek Ziadé | http://ziade.org

On Thu, Apr 16, 2009 at 12:54:02PM +0200, Tarek Ziadé wrote:
On Thu, Apr 16, 2009 at 12:42 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
Do these improvements sound sensible? And if so should I create one patch for each (and two bug reports) or combine them into one patch?
They sound reasonable, please fill two differents bugs with your patches and if you can, add some unit tests.
Attached are new tests for the distutils.unixccompiler module, testing the current behaviour of what I'm about to change (not testing the entire module!). Any comments on these tests would be appreciated, just to make sure I'm not doing it all wrong.
As long as you provide some support to test the final vesion on the various platforms (eg being a beta tester of the distutils trunk), I am able to work on your patches, and on the tests.
I should be able to test on some platforms (GNU/Linux-gcc, Solaris-gcc, AIX-gcc). Is the ditutils trunk still python 2.x trunk? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Thu, Apr 16, 2009 at 11:42:28AM +0100, Floris Bruynooghe wrote:
Now when using the --rpath option to the build_ext command distutils.unixcompiler.UnixCCompiler.runtime_library_dir_option() will use some heuristics to figure out which option to pass to compiler to get the runpath in (i.e. "-R" or "-Wl,-R" etc). I'm going to argue that this needs to be extened pass in -Wl,--enable-new-dtags,-R if the GNU linker is used so that the newer and better RUNPATH gets put into the shared objects all the time. (I don't yet know how to detect the GNU linker but would like a consensus on the desired behaviour before looking into this).
The patch for this is attached to http://bugs.python.org/issue5900 Feel free to give feedback/criticism if you care about this.
The second issue with build_ext --rpath is on AIX. Again some background on AIX shared objects, AIX is not SysV-like and uses the XCOFF binary format instead of ELF. Therefore they don't have a RPATH or RUNPATH, but they do have a think called LIBPATH which does something similar. The difference between XCOFF's LIBPATH and ELF's RUNPATH is that AIX's runtime linker does not have a default search path, hence the full search path needs to be encoded into the LIBPATH of the shared object.
Now I would propose for build_ext --rpath to encode the LIBPATH when used on AIX since that is the correct thing to do IMHO.
I was kind of hoping someone would pick up on this and at least argue about how it is not a good thing to try and convert an option for one concept (RPATH/RUNPATH) onto another one that isn't quite the same (LIBPATH). The side-effects are just not as clearly defined and open to interpretation. Anyway, having given this some more tought I now think it's not a good thing. Instead I'm going to make a patch that raises and exception if the --rpath option is used on AIX as it's behaviour is undefined[0]. If a user does want -blibpath:/some/path to be passed on to the linker they can still use LDFLAGS. Other opinions would be appreciated. Regards Floris [0] The linker will actually simply ignore it lurring the user into false security. Only when -bsvr4 is passed to the linker too will it be treated identical as -blibpath, but this is again the same fuzzy mapping as a SysV user will not be used to passing in the default search path to -R/--rpath and hence probably not end up with what they wanted too. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
participants (2)
-
Floris Bruynooghe
-
Tarek Ziadé