On Thu, Jun 24, 2010 at 11:27, Guido van Rossum <guido@python.org> wrote:
On Thu, Jun 24, 2010 at 10:48 AM, Brett Cannon <brett@python.org> wrote:
On Thu, Jun 24, 2010 at 08:50, Barry Warsaw <barry@python.org> wrote:
This is a follow up to PEP 3147. That PEP, already implemented in Python 3.2, allows for Python source files from different Python versions to live together in the same directory. It does this by putting a magic tag in the .pyc file name and placing the .pyc file in a __pycache__ directory.
Distros such as Debian and Ubuntu will use this to greatly simplifying deploying Python, and Python applications and libraries. Debian and Ubuntu usually ship more than one version of Python, and currently have to play complex games with symlinks to make this work. PEP 3147 will go a long way to eliminating the need for extra directories and symlinks.
One more thing I've found we need though, is a way to handled shared libraries for extension modules. Just as we can get name collisions on foo.pyc, we can get collisions on foo.so. We obviously cannot install foo.so built for Python 3.2 and foo.so built for Python 3.3 in the same location. So symlink nightmare's mini-me is back.
I have a fairly simple fix for this. I'd actually be surprised if this hasn't been discussed before, but teh Googles hasn't turned up anything.
The idea is to put the Python version number in the shared library file name, and extend .so lookup to find these extended file names. So for example, we'd see foo.3.2.so instead, and Python would know how to dynload both that and the traditional foo.so file too (for backward compatibility).
(On file naming: the original patch used foo.so.3.2 and that works just as well, but I thought there might be tools that expect exactly a '.so' suffix, so I changed it to put the Major.Minor version number to the left of the extension. The exact naming scheme is of course open to debate.)
While the idea is fine with me since I won't have any of my directories cluttered with multiple .so files, I would still want to add some moniker showing that the version number represents the interpreter and not the .so file. If I read "foo.3.2.so", that naively seems to mean to mean the foo module's 3.2 release is what is in installed, not that it's built for CPython 3.2. So even though it might be redundant, I would still want the VM name added.
Well, for versions of the .so itself, traditionally version numbers are appended *after* the .so suffix (check your /lib directory :-).
Second thing you taught me today (first was the x[:0] trick)! I've also been on OS X too long; /usr/lib is just .dynalib and that puts the version number before the extension.
Adding the VM name also doesn't make extension modules the exclusive domain of CPython either. If some other VM decides to make their own .so files that are not binary compatible then we should not preclude that as this solution it is nothing more than it makes a string comparison have to look at 7 more characters.
-Brett
P.S.: I wish we could drop use of the 'module.so' variant at the same time, for consistency sake and to cut out a stat call, but I know that is asking too much.
I wish so too. IIRC there used to be some modules that on Windows were wrappers around 3rd party DLLs and you can't have foo.dll as the module wrapping foo.dll the 3rd party DLL. (On Unix this problem doesn't exist because the 3rd party .so would be named libfoo.so, not foo.so.)
Wouldn't Barry's proposed solution actually fill this need since it will give the file a custom Python suffix that more-or-less guarantees no name clash with a third-party DLL?