Hello
Distutils allows you to use a handy --rpath option to the build_ext
command to add an RPATH value into the linked object. However some of
the semantics are not great IMHO.
For the first issue some background first. On SysV-like systems
(systems using the ELF binary format) the RPATH field was created in
the .dynamic section to allow the shared object to specify extra
locations to look for their required shared objects. However this did
overwrite the use of the LD_LIBRARY_PATH and that was no good for
administrators, so another new field was added: RUNPATH. This was
only used *after* LD_LIBRARY_PATH instead of before it solving the
problem. When both fields are present the runtime linker ignores
RPATH (it is impossible to create a shared object with a RUNPATH but
no RPATH).
The Solaris linker decided to always encode RUNPATH into the shared
object whenever an RPATH is specified (using the -R option). This is
a very sensible option most likely the correct behaviour. The GNU
linker however decided that backwards compatibility was more important
and to make it add the RUNPATH field you have to pass
--enable-new-dtags to the linker, if you use -R or -rpath on it's own
you will only get an RPATH, no RUNPATH.
Now when using the --rpath option to the build_ext command
distutils.unixcompiler.UnixCCompiler.runtime_library_dir_option() will
use some heuristics to figure out which option to pass to compiler to
get the runpath in (i.e. "-R" or "-Wl,-R" etc). I'm going to argue
that this needs to be extened pass in -Wl,--enable-new-dtags,-R if the
GNU linker is used so that the newer and better RUNPATH gets put into
the shared objects all the time. (I don't yet know how to detect the
GNU linker but would like a consensus on the desired behaviour before
looking into this).
Note that this is not completely a disaster, thanks to
distutils.sysconfig.customize_compiler() adding the contents of the
LDFLAGS environment variable to the command line of the linker
invocation you can work around this currently.
The second issue with build_ext --rpath is on AIX. Again some
background on AIX shared objects, AIX is not SysV-like and uses the
XCOFF binary format instead of ELF. Therefore they don't have a RPATH
or RUNPATH, but they do have a think called LIBPATH which does
something similar. The difference between XCOFF's LIBPATH and ELF's
RUNPATH is that AIX's runtime linker does not have a default search
path, hence the full search path needs to be encoded into the LIBPATH
of the shared object.
Now I would propose for build_ext --rpath to encode the LIBPATH when
used on AIX since that is the correct thing to do IMHO.
(Implementation note: since this is done by passing
-blibpath:/opt/example.com/lib:/usr/lib:/lib to the linker note that
distutils.unixccompiler.UnixCCompiler.link() would have to be changed
not to strip out /usr/lib and /lib from the runtime_library_dirs on
AIX. And
distutils.unixccompiler.UnixCCompiler.runtime_library_dir_option()
would have to use -blibpath:... or -Wl,-blibpath as appropriate)
Again, this can currently be circumvented by using the LDFLAGS
environment variable.
Do these improvements sound sensible? And if so should I create one
patch for each (and two bug reports) or combine them into one patch?
Regards
Floris
--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
I think that, going forward, Python packaging tools (not installation
tools; they should remain as they are, for backwards compatibility)
should move to supporting only One Package Per Project. And, each
project should have the same name as the package inside. In the future,
people should have to download an old copy of distutils deliberately if
they want to build projects with several packages inside; we should stop
releasing tools that support or encourage it.
1. It is easier on developers who want to "import escher" to know that
they can simply list "escher" as a dependency instead of having to
guess whether it's "Escher" or "EscherProject" or whether it's part
of a larger "lithographers" project or whether, heaven forbid, the
author decided to redundantly call the project "pyescher".
2. This practice would make PyPI's name make actual sense. It actually
claims to be (you can check the site!) the "Python *Package* Index"
whereas in fact it's currently nothing of the sort! It's really an
index of "projects" that might have zero, one, or several packages
inside of them. We should move all projects towards the good
behavior of the ones that already name themselves after the single
package that they contain.
3. I think the whole idea of putting several packages in a project was
useful back when dependencies didn't exist. It made sense, in
ancient days, for "ZODB" to include "transaction" because there was
no other way to make sure they got installed together. But now that
dependencies are possible, there is no longer a need for multiple-
package project that outweights the costs involved.
4. The current scheme makes it impossible to choose a "safe" package
name when creating and registering a new package. Just because
there's no "escher" *project* when you look at PyPI doesn't mean
that some project doesn't have an "escher" package hidden inside.
You could choose a package name, distribute your product, and only
find out later that your users cannot install both your product and
another product simultaneously because the other product was, in
fact, already using that package name but without your knowing it.
--
Brandon Craig Rhodes brandon(a)rhodesmill.org http://rhodesmill.org/brandon
I'm -1 on that, for three reasons.
1. I have a number of packages where the PyPI name is not the name of the toplevel package where the two names aren't the same on purpose. An example of this is "pyobjc-framework-WebKit", containing the python package "WebKit". The two don't have the same name because "import WebKit" is more natural during imports, while "pyobjc-framework-WebKit" is clearer in the PyPI listing (you don't have to wonder if this is some cross-platform web toolkit, it's obviously related to PyObjC).
2. For basicly the same reason a number of my PyPI packages have a number of toplevel Python packages. That's again because that makes sense for these packages, and it's furthermore needed for backward compatibility.
3. There actually a good reason for using "pysomelib" instead of "somelib" as the PyPI name for the python bindings for the C library "somelib". Naming the python bindings the same as the base project is confusion, while at the same time "import pysomelib" looks lame in Python code.
That said, I agree that there should be a good reason for not using the python package/module name as the PyPI name, and I'm +1 on adding advice to the distutils documentation to keep the two the same.
It's also not clear to me what your proposal would mean for namespace packages. Would packages like "zope.interface" be allowed with your proposal?
Ronald
On Friday, May 01, 2009, at 04:20PM, "Brandon Craig Rhodes" <brandon(a)rhodesmill.org> wrote:
>I think that, going forward, Python packaging tools (not installation
>tools; they should remain as they are, for backwards compatibility)
>should move to supporting only One Package Per Project. And, each
>project should have the same name as the package inside. In the future,
>people should have to download an old copy of distutils deliberately if
>they want to build projects with several packages inside; we should stop
>releasing tools that support or encourage it.
>
> 1. It is easier on developers who want to "import escher" to know that
> they can simply list "escher" as a dependency instead of having to
> guess whether it's "Escher" or "EscherProject" or whether it's part
> of a larger "lithographers" project or whether, heaven forbid, the
> author decided to redundantly call the project "pyescher".
>
> 2. This practice would make PyPI's name make actual sense. It actually
> claims to be (you can check the site!) the "Python *Package* Index"
> whereas in fact it's currently nothing of the sort! It's really an
> index of "projects" that might have zero, one, or several packages
> inside of them. We should move all projects towards the good
> behavior of the ones that already name themselves after the single
> package that they contain.
>
> 3. I think the whole idea of putting several packages in a project was
> useful back when dependencies didn't exist. It made sense, in
> ancient days, for "ZODB" to include "transaction" because there was
> no other way to make sure they got installed together. But now that
> dependencies are possible, there is no longer a need for multiple-
> package project that outweights the costs involved.
>
> 4. The current scheme makes it impossible to choose a "safe" package
> name when creating and registering a new package. Just because
> there's no "escher" *project* when you look at PyPI doesn't mean
> that some project doesn't have an "escher" package hidden inside.
> You could choose a package name, distribute your product, and only
> find out later that your users cannot install both your product and
> another product simultaneously because the other product was, in
> fact, already using that package name but without your knowing it.
>
>--
>Brandon Craig Rhodes brandon(a)rhodesmill.org http://rhodesmill.org/brandon
>_______________________________________________
>Distutils-SIG maillist - Distutils-SIG(a)python.org
>http://mail.python.org/mailman/listinfo/distutils-sig
>
>
