[Python-Dev] versioned .so files for Python 3.2

[Guido van Rossum]

On Thu, Jun 24, 2010 at 10:48 AM, Brett Cannon <brett at python.org> wrote:
> On Thu, Jun 24, 2010 at 08:50, Barry Warsaw <barry at python.org> wrote:
>> This is a follow up to PEP 3147.  That PEP, already implemented in Python 3.2,
>> allows for Python source files from different Python versions to live together
>> in the same directory.  It does this by putting a magic tag in the .pyc file
>> name and placing the .pyc file in a __pycache__ directory.
>>
>> Distros such as Debian and Ubuntu will use this to greatly simplifying
>> deploying Python, and Python applications and libraries.  Debian and Ubuntu
>> usually ship more than one version of Python, and currently have to play
>> complex games with symlinks to make this work.  PEP 3147 will go a long way to
>> eliminating the need for extra directories and symlinks.
>>
>> One more thing I've found we need though, is a way to handled shared libraries
>> for extension modules.  Just as we can get name collisions on foo.pyc, we can
>> get collisions on foo.so.  We obviously cannot install foo.so built for Python
>> 3.2 and foo.so built for Python 3.3 in the same location.  So symlink
>> nightmare's mini-me is back.
>>
>> I have a fairly simple fix for this.  I'd actually be surprised if this hasn't
>> been discussed before, but teh Googles hasn't turned up anything.
>>
>> The idea is to put the Python version number in the shared library file name,
>> and extend .so lookup to find these extended file names.  So for example, we'd
>> see foo.3.2.so instead, and Python would know how to dynload both that and the
>> traditional foo.so file too (for backward compatibility).
>>
>> (On file naming: the original patch used foo.so.3.2 and that works just as
>> well, but I thought there might be tools that expect exactly a '.so' suffix,
>> so I changed it to put the Major.Minor version number to the left of the
>> extension.  The exact naming scheme is of course open to debate.)
>>
>
> While the idea is fine with me since I won't have any of my
> directories cluttered with multiple .so files, I would still want to
> add some moniker showing that the version number represents the
> interpreter and not the .so file. If I read "foo.3.2.so", that naively
> seems to mean to mean the foo module's 3.2 release is what is in
> installed, not that it's built for CPython 3.2. So even though it
> might be redundant, I would still want the VM name added.

Well, for versions of the .so itself, traditionally version numbers
are appended *after* the .so suffix (check your /lib directory :-).

> Adding the VM name also doesn't make extension modules the exclusive
> domain of CPython either. If some other VM decides to make their own
> .so files that are not binary compatible then we should not preclude
> that as this solution it is nothing more than it makes a string
> comparison have to look at 7 more characters.
>
> -Brett
>
> P.S.: I wish we could drop use of the 'module.so' variant at the same
> time, for consistency sake and to cut out a stat call, but I know that
> is asking too much.

I wish so too. IIRC there used to be some modules that on Windows were
wrappers around 3rd party DLLs and you can't have foo.dll as the
module wrapping foo.dll the 3rd party DLL. (On Unix this problem
doesn't exist because the 3rd party .so would be named libfoo.so, not
foo.so.)

-- 
--Guido van Rossum (python.org/~guido)