[Python-Dev] versioned .so files for Python 3.2

[Barry Warsaw]

On Jun 30, 2010, at 10:35 PM, M.-A. Lemburg wrote:

>The Python default installation dir for
>libs (including site-packages) is $prefix/lib/pythonX.X, so you
>already have separate and properly versioned directory paths.
>
>What difference would the extra version on the .so file make in
>such a setup ?

So on Debian (inherted by Ubuntu), that would be /usr/lib/pythonX.Y and in
fact if you look there, you'll see the normal standard library layout you'd
expect.  There is no site-packages under there though because we install
add-ons to dist-packages[*] but functionally it's what you'd expect.

However, if you look inside dist-packages, you'll see something a little
weird.  All the .py files .egg-info files will actually be symlinks into a
parallel hierarchy under /usr/share/pyshared.  Also, under dist-packages
you'll see a directory layout that looks like it normally would under a
standard site-packages layout, except that again, all the .py files are
symlinked to the pyshared location.  The .so and .pyc files in dist-packages
will be the actual .so and .pyc files.

Why is that?  Well, it's because the .py files can be shared but the .pyc and
.so files can't.  Getting rid of these crufty symlink farms was the initial
motivation for PEP 3147.

In a PEP 3147 world, pure-Python packages don't need the directory
/usr/lib/pythonX.Y/dist-packages.  We can just install packages to
/usr/share/pyshared and none of the .pyc files will collide between Python
versions.  It makes distro packaging simpler (no symlinks to keep track of or
post-install linking and post-removal clean up to do) and smaller (only one
.py file for all supported Python versions).  All we have to do is a
post-install byte-compilation and we're done.  When we remove such a package
we only *have* to remove the .py source file because PEP 3147 will not load a
__pycache__ pyc if the source is missing (though it's easy enough to clean up
the pyc files too).

Throw extension modules into the mix and we have the file collision problem
again.  We can't install _foo.so into /usr/share/pyshared because that's going
to be Python ABI-specific.  If we can ABI-tag the .so and ensure it will get
imported, then we're back to the simpler layout where /usr/share/pyshared is
the destination for all installed Python packages.

-Barry

[*] That's done as a compromise between Debian's interpretation of the FHS and
the conflict of that interpretation to from-source installs of Python.
Specifically, Debian puts /usr/local/lib/pythonX.Y/dist-packages on the
*system* Python's path because that's where it expects system administrators
to put their own add-ons to the system Python.  That used to be
/usr/local/lib/pythonX.Y/site-packages but then if you did a from-source
altinstall for example of the same Python version, it would be possible for a
"/usr/local/bin/pythonX.Y setup.py install" of a third-party package to break
your system Python.  Not good!  (And yes, it happened to me :).

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100630/e5a489fe/attachment-0001.pgp>