2010/6/24 Barry Warsaw :

On Jun 24, 2010, at 10:58 AM, Benjamin Peterson wrote:

...

2010/6/24 Barry Warsaw :

...

Please let me know what you think.  I'm happy to just commit this to the py3k branch if there are no objections <wink>.  I don't think a new PEP is in order, but an update to PEP 3147 might make sense.

How will this interact with PEP 384 if that is implemented? I'm trying to come up with something that will work immediately while PEP 384 is being adopted.

But how will modules specify that they support multiple ABIs then? -- Regards, Benjamin

On Jun 24, 2010, at 02:28 PM, Barry Warsaw wrote:

On Jun 24, 2010, at 01:00 PM, Benjamin Peterson wrote:

...

2010/6/24 Barry Warsaw :

...

On Jun 24, 2010, at 10:58 AM, Benjamin Peterson wrote:

...

2010/6/24 Barry Warsaw :

...

Please let me know what you think.  I'm happy to just commit this to the py3k branch if there are no objections <wink>.  I don't think a new PEP is in order, but an update to PEP 3147 might make sense.

How will this interact with PEP 384 if that is implemented? I'm trying to come up with something that will work immediately while PEP 384 is being adopted.

But how will modules specify that they support multiple ABIs then?

I didn't understand, so asked Benjamin for clarification in IRC.

<gutworth> barry: if python 3.3 will only load x.3.3.so, but x.3.2.so supports the stable abi, will it load it? [14:25] <barry> gutworth: thanks, now i get it :) [14:26] <barry> gutworth: i think it should, but it wouldn't under my scheme. let me think about it

So, we could say that PEP 384 compliant extension modules would get written without a version specifier. IOW, we'd treat foo.so as using the ABI. It would then be up to the Python runtime to throw ImportErrors if in fact we were loading a legacy, non-PEP 384 compliant extension. -Barry

On Jun 26, 2010, at 10:22 PM, Matthias Klose wrote:

On 24.06.2010 22:46, Barry Warsaw wrote:

...

So, we could say that PEP 384 compliant extension modules would get written without a version specifier. IOW, we'd treat foo.so as using the ABI. It would then be up to the Python runtime to throw ImportErrors if in fact we were loading a legacy, non-PEP 384 compliant extension.

Is it realistic to never break the ABI? I would think of having the ABI encoded in the file name as well, and only bump the ABI if it does change. With the "versioned .so files" proposal an ABI bump is necessary with every python version, with PEP 384 the ABI bump will be decoupled from the python version.

You're right that the ABI will break, requiring a bump, and I think you're right that this means that PEP 384 compliant shared libraries would have to have a version number in their file name (assuming the versioned .so proposal is accepted). The problem is that we would need two version numbers, one for extension modules that are not PEP 384 complaint (and thus get bumped for every new Python version), and one for modules that are PEP 384 compliant (and thus only get bumped once in a while). The reason is that I think it will always be the case that we will have PEP 384 compliant and non-compliant extension modules. Perhaps identifying the underlying problems will lead to a more acceptable patch for Python. My patch tries to take a simple (perhaps too simplistic) solution, and I'm not married to it, but I think the general idea of versioned .so files is the right one. 1. The file name extensions that Python searches for are hardcoded and compiled in. dyload_shlib.c hard codes the file name pattern that extension modules must have in order for Python to load them. They must be <foo>.so or <foo>module.so. This gets compiled into Python at build time and there's no way for a distro (or anyone else who builds Python from source) to extend the file name patterns without modifying the source code. 2. The extension that distutils writes for shared libraries is dictated by build-time options and cannot be overridden. When you ./configure Python, autoconf figures out what shared library extension your platform uses. It substitutes this into a Makefile variable. That Makefile gets installed into your system with the base Python package and distutils parses the Makefile looking for this variable. When distutils calls your platform compiler, it uses this Makefile variable as the file name extension to use for your shared library. You cannot change this or override it to get distutils to write some other file name extension, well. Of these two problems, #1 is more serious because we have to modify the Python source code to hack in addition shared library search suffixes. #2 can be worked around by renaming the .so file after the build. The disadvantage of this though is that if you're a local packager, you'll have to remember to do the same thing if you want multiple Python version support, because distutils won't take care of it for you. Maybe that's okay, in which case it would still be good to address #1. -Barry

On Jun 24, 2010, at 11:05 AM, Daniel Stutzbach wrote:

On Thu, Jun 24, 2010 at 10:50 AM, Barry Warsaw wrote:

...

The idea is to put the Python version number in the shared library file name, and extend .so lookup to find these extended file names. So for example, we'd see foo.3.2.so instead, and Python would know how to dynload both that and the traditional foo.so file too (for backward compatibility).

What use case does this address?

Specifically, it's the use case where we (Debian/Ubuntu) plan on installing all Python 3.x packages into /usr/lib/python3/dist-packages. As of PEP 3147, we can do that without collisions on the pyc files, but would still have to symlink for extension module .so files, because they are always named foo.so and Python 3.2's foo.so won't (modulo PEP 384) be compatible with Python 3.3's foo.so. So using the same trick as in PEP 3147, if we can name Python 3.2's foo extension differently than the incompatible Python 3.3's foo extension, we can have them live in the same directory without symlink tricks.

PEP 3147 addresses the fact that the user may have different versions of Python installed and each wants to write a .pyc file when loading a module. .so files are not generated simply by running the Python interpreter, ergo .so files are not an issue for that use case.

See above. It doesn't matter whether the pyc or so is created at run time by the user or by the distro build system. If the files for different Python versions end up in the same directory, they must be named differently too.

If you want to make it so a system can install a package in just one location to be used by multiple Python installations, then the version number isn't enough. You also need to distinguish debug builds, profiling builds, Unicode width (see issue8654), and probably several other ./configure options.

This is a good point, but more easily addressed. Let's say a distro makes three Python 3.2 variants available, one "normal" build, a debug build, and UCS2 and USC4 versions of the above. All we need to do is choose a different .so ABI tag (see previous follow) for each of those builds. My updated patch (coming soon) allows you to define that tag to configure. So e.g. Normal build UCSX: SOABI=cpython-32 ./configure Debug build UCSX: SOABI=cpython-32-d ./configure Normal build UCSY: SOABI=cpython-32-w ./configure Debug build UCSY: SOABI=cpython-32-dw ./configure Mix and match for any other build options you care about. Because the distro controls how Python is configured, this should be fairly easy to achieve. -Barry

Scott Dial wrote:

On 6/24/2010 5:09 PM, Barry Warsaw wrote:

...

...

What use case does this address?

...

If you want to make it so a system can install a package in just one location to be used by multiple Python installations, then the version number isn't enough. You also need to distinguish debug builds, profiling builds, Unicode width (see issue8654), and probably several other ./configure options.

This is a good point, but more easily addressed. Let's say a distro makes three Python 3.2 variants available, one "normal" build, a debug build, and UCS2 and USC4 versions of the above. All we need to do is choose a different .so ABI tag (see previous follow) for each of those builds. My updated patch (coming soon) allows you to define that tag to configure. So e.g.

Why is this use case not already addressed by having independent directories? And why is there an incentive to co-mingle these version-punned files with version-agnostic ones?

I don't think this is a good idea. After a while your Python lib directories would need some serious dusting off to make them maintainable again. Disk space is cheap so setting up dedicated directories for each variant will result in a much easier to manage installation. If you want a really clever setup, use hard links between those directory (you can also use symlinks if you like). Then a change in one Python file will automatically propagate to all other variant dirs without any maintenance effort. Together with PYTHONHOME this makes a really nice virtualenv-like environment. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 25 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-07-19: EuroPython 2010, Birmingham, UK 23 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Barry Warsaw wrote:

On Jun 25, 2010, at 12:35 AM, M.-A. Lemburg wrote:

...

Scott Dial wrote:

...

On 6/24/2010 5:09 PM, Barry Warsaw wrote:

...

...

What use case does this address?

...

If you want to make it so a system can install a package in just one location to be used by multiple Python installations, then the version number isn't enough. You also need to distinguish debug builds, profiling builds, Unicode width (see issue8654), and probably several other ./configure options.

This is a good point, but more easily addressed. Let's say a distro makes three Python 3.2 variants available, one "normal" build, a debug build, and UCS2 and USC4 versions of the above. All we need to do is choose a different .so ABI tag (see previous follow) for each of those builds. My updated patch (coming soon) allows you to define that tag to configure. So e.g.

Why is this use case not already addressed by having independent directories? And why is there an incentive to co-mingle these version-punned files with version-agnostic ones?

I don't think this is a good idea. After a while your Python lib directories would need some serious dusting off to make them maintainable again.

Disk space is cheap so setting up dedicated directories for each variant will result in a much easier to manage installation.

If you want a really clever setup, use hard links between those directory (you can also use symlinks if you like). Then a change in one Python file will automatically propagate to all other variant dirs without any maintenance effort. Together with PYTHONHOME this makes a really nice virtualenv-like environment.

Note that I do believe there is a difference between what users maintaining their own Python installations might want, and what a distro needs to maintain its entire Python stack. So while dedicated directories might make more sense if you're maintaining your own Python built from source, it doesn't make as much sense for a distro, as described in previous responses by Matthias.

Fair enough. I haven't followed the thread closely, so Matthias will probably already have answered this: The Python default installation dir for libs (including site-packages) is $prefix/lib/pythonX.X, so you already have separate and properly versioned directory paths. What difference would the extra version on the .so file make in such a setup ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 30 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-07-19: EuroPython 2010, Birmingham, UK 18 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Jun 24, 2010, at 5:53 PM, Scott Dial wrote:

On 6/24/2010 5:09 PM, Barry Warsaw wrote:

...

...

What use case does this address?

Specifically, it's the use case where we (Debian/Ubuntu) plan on installing all Python 3.x packages into /usr/lib/python3/dist-packages. As of PEP 3147, we can do that without collisions on the pyc files, but would still have to symlink for extension module .so files, because they are always named foo.so and Python 3.2's foo.so won't (modulo PEP 384) be compatible with Python 3.3's foo.so.

If the package has .so files that aren't compatible with other version of python, then what is the motivation for placing that in a shared location (since it can't actually be shared)

Because python looks for .so files in the same place it looks for the .py files of the same package. E.g., given a module like lxml, it contains the following files (among others): lxml/ lxml/__init__.py lxml/__init__.pyc lxml/builder.py lxml/builder.pyc lxml/etree.so And you can only put it in one place. Really, python should store the .py files in /usr/share/python/, the .so files in /usr/lib/x86_64- linux-gnu/python2.5-debug/, and the .pyc files in /var/lib/python2.5- debug. But python doesn't work like that. James

On 25.06.2010 02:54, Ben Finney wrote:

James Y Knight writes:

...

Really, python should store the .py files in /usr/share/python/, the .so files in /usr/lib/x86_64- linux-gnu/python2.5-debug/, and the .pyc files in /var/lib/python2.5- debug. But python doesn't work like that.

+1

So who's going to draft the “Filesystem Hierarchy Standard compliance” PEP? :-)

This has nothing to do with the FHS. The FHS talks about data, not code.

On 26.06.2010 22:30, C. Titus Brown wrote:

On Sat, Jun 26, 2010 at 10:25:28PM +0200, Matthias Klose wrote:

...

On 25.06.2010 02:54, Ben Finney wrote:

...

James Y Knight writes:

...

Really, python should store the .py files in /usr/share/python/, the .so files in /usr/lib/x86_64- linux-gnu/python2.5-debug/, and the .pyc files in /var/lib/python2.5- debug. But python doesn't work like that.

+1

So who's going to draft the ???Filesystem Hierarchy Standard compliance??? PEP? :-)

This has nothing to do with the FHS. The FHS talks about data, not code.

Really? It has some guidelines here for object files, etc., at least as of 2004.

http://www.pathname.com/fhs/pub/fhs-2.3.html

A quick scan suggests /usr/lib is the right place to look:

http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGAN...

agreed for object files, but http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENT... explicitely states "The /usr/share hierarchy is for all read-only architecture independent *data* files".

On 6/24/2010 8:23 PM, James Y Knight wrote:

On Jun 24, 2010, at 5:53 PM, Scott Dial wrote:

...

If the package has .so files that aren't compatible with other version of python, then what is the motivation for placing that in a shared location (since it can't actually be shared)

Because python looks for .so files in the same place it looks for the .py files of the same package.

My suggestion was that a package that contains .so files should not be shared (e.g., the entire lxml package should be placed in a version-specific path). The motivation for this PEP was to simplify the installation python packages for distros; it was not to reduce the number of .py files on the disk. Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way. You must still compile the package multiple times for each relevant version of python (with special tagging that I imagine distutils can take care of) and, worse yet, you have created a more trick install than merely having multiple search paths (e.g., installing/uninstalling lxml for *one* version of python is actually more difficult in this scheme). Either the motivation for this PEP is inaccurate or I am failing to understand how this is *simpler*. In the case of pure-python, this PEP is clearly a win, but I have not seen an argument that it is a win for .so files. Moreover, the PEP itself is titled "PYC Repository Directories" (not "shared site-packages") and makes no mention of .so files at all. -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

On 6/25/2010 2:58 PM, Brett Cannon wrote:

I assume you are talking about PEP 3147. You're right that the PEP was for pyc files and that's it. No one is talking about rewriting the PEP.

Yes, I am making reference to PEP 3147. I make reference to that PEP because this change is of the same order of magnitude as the .pyc change, and we asked for a PEP for that, and if this .so stuff is an extension of that thought process, then it should either be reflected by that PEP or a new PEP.

The motivation Barry is using is an overarching one of distros wanting to use a single directory install location for all installed Python versions. That led to PEP 3147 and now this work.

It's unclear to me that that is the correct motivation, which you are divining. As I understand it, the motivation to be to *simplify installation* for distros, which may or may not be achieved by using a single directory. In the case of pure-python packages, a single directory is an obvious win. In the case of mixed-python packages, I remain to be persuaded there is any improvement achieved.

This is meant to be used by distros in a programmatic fashion, so my response is "so what?" Their package management system is going to maintain the directory, not a person.

Then why is the status quo unacceptable? I have already explained how this will still require programmatic steps of at least the same difficulty as the status quo requires, so why should we change anything? I am skeptical that this is a simple programmatic problem either: take any random package on PyPI and tell me whether or not it has a .so file that must be compiled. If such a .so file exists, then this package must be special-cased and compiled for each version of Python on the system (or will ever be on the system?). Such a package yields an arbitrary number of .so files due to the number of version of Python on the machine, and I can't imagine how it is simpler to manage all of those files than it is to manage multiple site-packages.

You're conflating what is being discussed with PEP 3147. That PEP is independent of this. PEP 3147 just empowered this work to be relevant.

Without a PEP (be it PEP 3147 or some other), what is the justification for doing this? The burden should be on "you" to explain why this is a good idea and not just a clever idea. -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

On 25.06.2010 20:58, Brett Cannon wrote:

On Fri, Jun 25, 2010 at 01:53, Scott Dial

...

Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way. You must still compile the package multiple times for each relevant version of python (with special tagging that I imagine distutils can take care of) and, worse yet, you have created a more trick install than merely having multiple search paths (e.g., installing/uninstalling lxml for *one* version of python is actually more difficult in this scheme).

This is meant to be used by distros in a programmatic fashion, so my response is "so what?" Their package management system is going to maintain the directory, not a person. You and I are not going to be using this for anything. This is purely meant for Linux OS vendors (maybe OS X) to manage their installs through their package software. I honestly do not expect human beings to be mucking around with these installs (and I suspect Barry doesn't either).

Placing files for a distribution in a version-independent path does help distributions handling file conflicts, detecting duplicates and with moving files between different (distribution) packages. Having non-conflicting extension names is a schema which already is used on some platforms (debug builds on Windows). The question for me is, if just a renaming of the .so files is acceptable for upstream, or if distributors should implement this on their own, as something like: if ext_path.startswith('/usr/') and not ext_path.startswith('/usr/local/'): load_ext('foo.2.6.so') else: load_ext('foo.so') I fear this will cause issues when e.g. virtualenv environments start copying parts from the system installation instead of symlinking it. Matthias

On Jun 25, 2010, at 11:58 AM, Brett Cannon wrote:

...

Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way. You must still compile the package multiple times for each relevant version of python (with special tagging that I imagine distutils can take care of) and, worse yet, you have created a more trick install than merely having multiple search paths (e.g., installing/uninstalling lxml for *one* version of python is actually more difficult in this scheme).

This is meant to be used by distros in a programmatic fashion, so my response is "so what?" Their package management system is going to maintain the directory, not a person. You and I are not going to be using this for anything. This is purely meant for Linux OS vendors (maybe OS X) to manage their installs through their package software. I honestly do not expect human beings to be mucking around with these installs (and I suspect Barry doesn't either).

Spot on. -Barry

On Sat, Jun 26, 2010 at 6:12 AM, James Y Knight wrote:

However, then you have to also consider python packages made up of multiple distro packages -- like twisted or zope. Twisted includes some C extensions in the core package. But then there are other twisted modules (installed under a "twisted.foo" name) which do not include C extensions. If the base twisted package is installed under a version-specific directory, then all of the submodule packages need to also be installed under the same version-specific directory (and thus built for all versions).

In the past, it has proven somewhat tricky to coordinate which directory the modules for package "foo" should be installed in, because you need to know whether *any* of the related packages includes a native ".so" file, not just the current package.

The converse situation, where a base package did *not* get installed into a version-specific directory because it includes no native code, but a submodule *does* include a ".so" file, is even trickier.

I think there are two major ways to tackle this: - allow multiple versions of a .so file within a single directory (i.e Barry's current suggestion) - enhanced namespace packages, allowing a single package to be spread across multiple directories, some of which may be Python version specific (i.e. modifications to PEP 382 to support references to version-specific directories) I think a new PEP is definitely in order, especially to explain why enhancing PEP 382 to support saying "look over here for the .so files for this version" isn't a preferable approach. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25.06.2010 22:12, James Y Knight wrote:

On Jun 25, 2010, at 4:53 AM, Scott Dial wrote:

...

On 6/24/2010 8:23 PM, James Y Knight wrote:

...

On Jun 24, 2010, at 5:53 PM, Scott Dial wrote:

...

If the package has .so files that aren't compatible with other version of python, then what is the motivation for placing that in a shared location (since it can't actually be shared)

Because python looks for .so files in the same place it looks for the .py files of the same package.

My suggestion was that a package that contains .so files should not be shared (e.g., the entire lxml package should be placed in a version-specific path). The motivation for this PEP was to simplify the installation python packages for distros; it was not to reduce the number of .py files on the disk.

Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way.

This is a good point, but I think still falls short of a solution. For a package like lxml, indeed you are correct. Since debian needs to build it once per version, it could just put the entire package (.py files and .so files) into a different per-python-version directory.

This is what is currently done. This will increase the size of packages by duplicating the .py files, or you have to install the .py in a common location (irrelevant to sys.path), and provide (sym)links to the expected location. A "different per-python-version directory" also has the disadvantage that file conflicts between (distribution) packages cannot be detected.

However, then you have to also consider python packages made up of multiple distro packages -- like twisted or zope. Twisted includes some C extensions in the core package. But then there are other twisted modules (installed under a "twisted.foo" name) which do not include C extensions. If the base twisted package is installed under a version-specific directory, then all of the submodule packages need to also be installed under the same version-specific directory (and thus built for all versions).

In the past, it has proven somewhat tricky to coordinate which directory the modules for package "foo" should be installed in, because you need to know whether *any* of the related packages includes a native ".so" file, not just the current package.

The converse situation, where a base package did *not* get installed into a version-specific directory because it includes no native code, but a submodule *does* include a ".so" file, is even trickier.

I don't think that installation into different locations based on the presence of extension will work. Should a location really change if an extension is added as an optimization? Splitting a (python) package into different installation locations should be avoided. Matthias

On Jun 26, 2010, at 06:50 PM, Scott Dial wrote:

On 6/26/2010 4:06 PM, Matthias Klose wrote:

...

On 25.06.2010 22:12, James Y Knight wrote:

...

On Jun 25, 2010, at 4:53 AM, Scott Dial wrote:

...

Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way.

This is a good point, but I think still falls short of a solution. For a package like lxml, indeed you are correct. Since debian needs to build it once per version, it could just put the entire package (.py files and .so files) into a different per-python-version directory.

This is what is currently done. This will increase the size of packages by duplicating the .py files, or you have to install the .py in a common location (irrelevant to sys.path), and provide (sym)links to the expected location.

"This is what is currently done" and "provide (sym)links to the expected location" are conflicting statements.

I think Matthias was referring to "what is currently done" to your statement "debian needs to build it once per version". Providing symlinks is how we are able to make it appear that there are version-specific py files without actually doing so.

If you are symlinking .py files from a shared location, then that is not the same as "just install the package into a version-specific location". What motivation is there for preferring symlinks?

This reduces .py file duplications in distro packages.

Who cares if a ditro package install yields duplicate .py files? Nor am I motivated by having to carry duplicate .py files in a distribution package (I imagine the compression of duplicate .py files is amazing).

It might be amazing, but it's still a significant overhead. As I've described, multiply that by all the py files in all the distro packages containing Python source code, and then still try to fit it on a CDROM.

What happens to the distro packaging if a python package splits the codebase between 2.x and 3.x (meaning they have distinct .py files)?

The Debian/Ubuntu approach to Python 2/3 support is to provide them in separate distro packages. E.g. for Python package foo, you would have Debuntu package python-foo (for the Python 2.x version) and python3-foo. We do not share source between Python 2 and 3 versions, at least not yet <wink>. This doesn't hurt us much because the number of Python packages that are source compatible between the two is pretty low (Benjamin's 'six' package might change that :), and not much depends on Python 3 yet.

As someone else mentioned, how is virtualenv going to interact with packages that install like this?

This is a good question, but I *think* it won't affect it much at all. To test for sure I'd either need a Python 3 compatible virtualenv or backport my patch to Python 2.6 and 2.7. But still, I'm not sure it would matter since the same shared library import suffix is used in either case. I actually think version-specific search paths would have a greater impact on virtualenv. -Barry

On Jul 01, 2010, at 07:02 AM, Scott Dial wrote:

I decided to prove to myself that it was not a significant issue to have parallel directory structures in a .tar.bz2, and I was surprised to find it much worse at that then I had imagined. For example,

# cd /usr/lib/python2.6/site-packages # tar --exclude="*.pyc" --exclude="*.pyo" \ -cjf mercurial.tar.bz2 mercurial # du -h mercurial.tar.bz2 640K mercurial.tar.bz2

# cp -a mercurial mercurial2 # tar --exclude="*.pyc" --exclude="*.pyo" \ -cjf mercurial2.tar.bz2 mercurial mercurial2 # du -h mercurial.tar.bz2 1.3M mercurial2.tar.bz2

So, I was definitely wrong in saying that you do better than doubling. [...] I appreciate all your replies. I am not sure a PEP is really needed here, but to having had all of this discussed and explained on the mailing list is certainly useful. I trust that yourself and the debuntu python group will end up chasing down and taking care of any quirks that this change might cause, so I am not worried about it. :D

Getting back to this after the US holiday. Thanks for running these numbers Scott. I've opened a bug in the Python tracker and attached my latest patch: http://bugs.python.org/issue9193 The one difference from previous versions of the patch is that the .so tag is now settable via "./configure --with-so-abi-tag=foo". This would generate shared libs like _multiprocessing.foo.so. I'd like to get consensus as to whether folks feel that a PEP is needed. My own thought is that I'd rather not do a PEP specific to this change, but I would update PEP 384 with the implications on .so versioning. Please also feel free to review the patch in that issue. Thanks, -Barry

Am 07.07.2010 23:04, schrieb Georg Brandl:

Am 07.07.2010 20:40, schrieb Barry Warsaw:

...

Getting back to this after the US holiday. Thanks for running these numbers Scott. I've opened a bug in the Python tracker and attached my latest patch:

http://bugs.python.org/issue9193

The one difference from previous versions of the patch is that the .so tag is now settable via "./configure --with-so-abi-tag=foo". This would generate shared libs like _multiprocessing.foo.so.

I'd like to get consensus as to whether folks feel that a PEP is needed. My own thought is that I'd rather not do a PEP specific to this change, but I would update PEP 384 with the implications on .so versioning. Please also feel free to review the patch in that issue.

I can see where this is going... writing it into PEP 384 would automatically get the change accepted?

I hit "Send" prematurely. I wanted to add that I'd be okay with this change, be it in a new PEP or an old one. Georg

On Thu, Jul 8, 2010 at 4:40 AM, Barry Warsaw wrote:

I'd like to get consensus as to whether folks feel that a PEP is needed.  My own thought is that I'd rather not do a PEP specific to this change, but I would update PEP 384 with the implications on .so versioning.  Please also feel free to review the patch in that issue.

I suspect you could write a new PEP faster than you could convince those suggesting the change needs a PEP (including me) that one isn't necessary. Presumably you were going to do a summary email for the mailing list anyway - just tidy up the formatting a bit and check it in as a PEP instead :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jul 08, 2010, at 01:47 AM, Matthias Klose wrote:

On 07.07.2010 20:40, Barry Warsaw wrote:

...

Getting back to this after the US holiday. Thanks for running these numbers Scott. I've opened a bug in the Python tracker and attached my latest patch:

http://bugs.python.org/issue9193

The one difference from previous versions of the patch is that the .so tag is now settable via "./configure --with-so-abi-tag=foo". This would generate shared libs like _multiprocessing.foo.so.

- imo, it's wrong to lookup _multiprocessing.so first, before looking up _multiprocessing.foo.so (at least for the use case to put the extensions for multiple python versions into one directory).

Good point.

- why is the flexibility of specifying the "foo" needed? The naming for the __pycache__ files is fixed, why have it configurable for extensions?

The 'foo' part in the shared library name is equivalent to the <tag> part in __pycache__/baz.<tag>.pyc, not specifically the __pycache__ part. Specifying the <tag> is necessary because extension modules built for Python 3.2 will not be compatible with extension modules built for Python 3.3 (in the absence of PEP 384), but they will live in the same directory. -Barry

On Jun 25, 2010, at 04:53 AM, Scott Dial wrote:

My suggestion was that a package that contains .so files should not be shared (e.g., the entire lxml package should be placed in a version-specific path).

Matthias outlined some of the pitfalls with this approach.

The motivation for this PEP was to simplify the installation python packages for distros; it was not to reduce the number of .py files on the disk.

As others have pointed out, versioned so files is not part of PEP 3147. That PEP does reduce the number of py files on disk, which as I explained in a previous follow, is an important consideration.

Placing .so files together does not simplify that install process in any way.

I disagree of course. :)

You will still have to handle such packages in a special way. You must still compile the package multiple times for each relevant version of python (with special tagging that I imagine distutils can take care of) and, worse yet,

No, distutils cannot take care of this. There is no way currently to tell distutils to generate a .so file with anything but the platform-specific way of spelling "shared library".

you have created a more trick install than merely having multiple search paths (e.g., installing/uninstalling lxml for *one* version of python is actually more difficult in this scheme).

That's not a use case we care about. If you have Python 3.2 and 3.3 installed on your system, why would you want lxml installed for one but not the other? And even if for some reason you did, the only way to do that would be in a way similar to handling the PEP 3147 pyc files. -Barry

On 6/24/2010 9:18 PM, Greg Ewing wrote:

Scott Dial wrote:

...

But the only motivation for doing this with .pyc files is that the .py files are able to be shared,

In an application made up of a mixture of pure Python and extension modules, the .py files are able to be shared too. Seems to me that a similar motivation exists here as well. Not exactly the same, but closely related.

If I recall Barry's motivation correctly, the PEP was intended to simplify the installation of packages for multiple versions of Python, although the PEP states that in a less direct way. In the case of pure-python packages, this is merely about avoiding .pyc collisions. But, in the case of packages with .so files, I fail to see how this is simpler (in face, I believe it to be more complicated). So, I am not sure the PEP supports this feature being proposed (since it makes no mention of .so files), and more importantly, I am not sure it actually makes anything better for anyone (still requires multiple compilations and un/install gymnastics). -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

On Thu, Jun 24, 2010 at 11:27, Guido van Rossum wrote:

On Thu, Jun 24, 2010 at 10:48 AM, Brett Cannon wrote:

...

On Thu, Jun 24, 2010 at 08:50, Barry Warsaw wrote:

...

This is a follow up to PEP 3147.  That PEP, already implemented in Python 3.2, allows for Python source files from different Python versions to live together in the same directory.  It does this by putting a magic tag in the .pyc file name and placing the .pyc file in a __pycache__ directory.

Distros such as Debian and Ubuntu will use this to greatly simplifying deploying Python, and Python applications and libraries.  Debian and Ubuntu usually ship more than one version of Python, and currently have to play complex games with symlinks to make this work.  PEP 3147 will go a long way to eliminating the need for extra directories and symlinks.

One more thing I've found we need though, is a way to handled shared libraries for extension modules.  Just as we can get name collisions on foo.pyc, we can get collisions on foo.so.  We obviously cannot install foo.so built for Python 3.2 and foo.so built for Python 3.3 in the same location.  So symlink nightmare's mini-me is back.

I have a fairly simple fix for this.  I'd actually be surprised if this hasn't been discussed before, but teh Googles hasn't turned up anything.

The idea is to put the Python version number in the shared library file name, and extend .so lookup to find these extended file names.  So for example, we'd see foo.3.2.so instead, and Python would know how to dynload both that and the traditional foo.so file too (for backward compatibility).

(On file naming: the original patch used foo.so.3.2 and that works just as well, but I thought there might be tools that expect exactly a '.so' suffix, so I changed it to put the Major.Minor version number to the left of the extension.  The exact naming scheme is of course open to debate.)

While the idea is fine with me since I won't have any of my directories cluttered with multiple .so files, I would still want to add some moniker showing that the version number represents the interpreter and not the .so file. If I read "foo.3.2.so", that naively seems to mean to mean the foo module's 3.2 release is what is in installed, not that it's built for CPython 3.2. So even though it might be redundant, I would still want the VM name added.

Well, for versions of the .so itself, traditionally version numbers are appended *after* the .so suffix (check your /lib directory :-).

Second thing you taught me today (first was the x[:0] trick)! I've also been on OS X too long; /usr/lib is just .dynalib and that puts the version number before the extension.

...

Adding the VM name also doesn't make extension modules the exclusive domain of CPython either. If some other VM decides to make their own .so files that are not binary compatible then we should not preclude that as this solution it is nothing more than it makes a string comparison have to look at 7 more characters.

-Brett

P.S.: I wish we could drop use of the 'module.so' variant at the same time, for consistency sake and to cut out a stat call, but I know that is asking too much.

I wish so too. IIRC there used to be some modules that on Windows were wrappers around 3rd party DLLs and you can't have foo.dll as the module wrapping foo.dll the 3rd party DLL. (On Unix this problem doesn't exist because the 3rd party .so would be named libfoo.so, not foo.so.)

Wouldn't Barry's proposed solution actually fill this need since it will give the file a custom Python suffix that more-or-less guarantees no name clash with a third-party DLL?

On Thu, Jun 24, 2010 at 4:55 PM, Barry Warsaw wrote:

Which is probably another reason not to use foo.so.X.Y for Python extension modules.

Clearly, foo.so.3.2 is the man page for the foo.so.3 system call. The ABI ident definitely has to be elsewhere. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "A storm broke loose in my mind." --Albert Einstein

On Thu, Jun 24, 2010 at 11:53, Éric Araujo wrote:

Le 24/06/2010 19:48, Brett Cannon a écrit :

...

P.S.: I wish we could drop use of the 'module.so' variant at the same time, for consistency sake and to cut out a stat call, but I know that is asking too much.

At least, looking for spam/__init__module.so could be avoided. It seems to me that the package definition does not allow that.

I thought no one had bothered to change import.c to allow for extension modules to act as a package's __init__? As for not being allowed, I don't agree with that assessment. If you treat a package's __init__ module as simply that, a module that would be named __init__ when imported, then __init__module.c would be valid (and that's what importlib does).

The tradeoff would be code complication for one less stat call. Worth a bug report?

Nah.

Your plan seems good. Adding keyword arguments should not create compatibility issues, and I suspect the impact on the code of build_ext may be actually quite small. I’ll try to review your patch even though I don’t know C or compiler oddities, but Tarek will have the best insight and the final word. In case the time machine’s not available, your suggestion about getting the filename from the Extension instance instead of passing in a string can most certainly land in distutils2. Regards

On Jun 25, 2010, at 08:35 AM, Nick Coghlan wrote:

I like the idea, but I think summarising the rest of this discussion in its own (relatively short) PEP would be good (there are a few things that are tricky - exact versioning scheme, PEP 384 forward compatibility, impact on distutils, articulating the benefits for distro packaging, etc).

The first draft of PEP 3149 is ready for review. http://www.python.org/dev/peps/pep-3149/ Plain text attached here for your convenience. Comments, suggestions as always are welcome. Thanks to everyone who participated in the original discussion. -Barry PEP: 3149 Title: ABI version tagged .so files Version: $Revision: 81577 $ Last-Modified: $Date: 2010-05-27 19:54:25 -0400 (Thu, 27 May 2010) $ Author: Barry Warsaw Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2010-07-09 Python-Version: 3.2 Post-History: 2010-07-14 Resolution: TBD Abstract ======== PEP 3147 [1]_ described an extension to Python's import machinery that improved the sharing of Python source code, by allowing more than one byte compilation file (.pyc) to be co-located with each source file. This PEP defines an adjunct feature which allows the co-location of extension module files (.so) in a similar manner. This optional, build-time feature will enable downstream distributions of Python to more easily provide more than one Python major version at a time. Background ========== PEP 3147 defined the file system layout for a pure-Python package, where multiple versions of Python are available on the system. For example, where the `alpha` package containing source modules `one.py` and `two.py` exist on a system with Python 3.2 and 3.3, the post-byte compilation file system layout would be:: alpha/ __pycache__/ __init__.cpython-32.pyc __init__.cpython-33.pyc one.cpython-32.pyc one.cpython-33.pyc two.cpython-32.pyc two.cpython-33.pyc __init__.py one.py two.py For packages with extension modules, a similar differentiation is needed for the module's .so files. Extension modules compiled for different Python major versions are incompatible with each other due to changes in the ABI. While PEP 384 [2]_ defines a stable ABI, it will minimize, but not eliminate extension module incompatibilities between Python major versions. Thus a mechanism for discriminating extension module file names is proposed. Rationale ========= Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more than one Python version at the same time to their users. For example, Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, with Python 2.6 being the default. In order to share as much as possible between the available Python versions, these distributions install third party (i.e. non-standard library) packages into `/usr/share/pyshared` and symlink to them from `/usr/lib/pythonX.Y/dist-packages`. The symlinks exist because in a pre-PEP 3147 world (i.e < Python 3.2), the `.pyc` files resulting from byte compilation by the various installed Pythons will name collide with each other. For Python versions >= 3.2, all pure-Python packages can be shared, because the `.pyc` files will no longer cause file system naming conflicts. Eliminating these symlinks makes for a simpler, more robust Python distribution. A similar situation arises with shared library extensions. Because extension modules are typically named `foo.so` for a `foo` extension module, these would also name collide if `foo` was provided for more than one Python version. There are several approaches that could be taken to avoid this, which will be explored below, but this PEP proposes a fairly simple compile-time option to allow extension modules to live in the same file system directory and avoid any name collisions. Proposal ======== A new configure option is added for building Python, called `--with-so-abi-tag`. This takes as an argument a unique, but arbitrary string, e.g.:: ./configure --with-so-abi-tag=cpython-32 This string is passed into the `Makefile` and affects two aspects of the Python build. First, it is compiled into `Python/dynload_shlib.c` where it defines some additional `.so` file names to search for when importing extension modules. Second, it modifies the `Makefile`'s `$SO` variable, which in turn controls the `distutils` module's default filename when compiling extension modules. When `--with-so-abi-tag` is not given to `configure` nothing changes in the way the Python executable is built, or acts. Thus, this configure switch is completely optional and has no effect if not used. What this allows is for distributions that want to distinguish among extension modules built for different versions of Python, but shared in the same file system path, to arrange for `.so` names that are unique and non-colliding. For example, let's say Python 3.2 was built with:: ./configure --with-so-abi-tag=cpython-32 and Python 3.3 was built with:: ./configure --with-so-abi-tag=cpython-33 For an arbitrary package `foo`, you would see these files when the distribution package was installed:: /usr/share/pyshared/foo.cpython-32.so /usr/share/pyshared/foo.cpython-33.so Proven approach =============== The approach described here is already proven, in a sense, on Debian and Ubuntu system where different extensions are used for debug builds of Python and extension modules. Debug builds on Windows also already use a different file extension for dynamic libraries. PEP 384 ======= PEP 384 defines a stable ABI for extension modules. Universal adoption of PEP 384 would eliminate the need for this PEP because all extension modules would be compatible with any Python version. In practice of course, it will be impossible to achieve universal adoption. Older extensions may not be ported to PEP 384, or an extension may require Python APIs outside of PEP 384 definition. Therefore there will always be a (hopefully diminishing, but never zero) need for ABI version tagged shared libraries. Further, it is anticipated that the stable ABI will evolve over time, meaning that existing PEP 384 compatible extension modules may be incompatible with future versions of Python. While a complete specification is reserved for PEP 384, here is a discussion of the relevant issues. PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is passed as the API version if the extension was complied with ``Py_LIMITED_API``. This should be formalized into an official macro called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``. If and when the ABI changes in an incompatible way, this version number would be bumped. To facilitate sharing, Python would be extended to search for extension modules with the ``PYTHON_ABI_VERSION`` number in its name. The prefix ``abi`` is reserved for Python's use. Thus for example, an initial implementation of PEP 384, compiled with `--with-so-abi-tag=cpython-xy` would search for the following file names when extension module `foo` is imported (in this order):: foo.abi3.so foo.cpython-xy.so foo.so The distutils [7]_ ``build_ext`` command would also have to be extended to compile to shared library files with the ``abi3`` tag, when the module author indicates that their extension supports that version of the ABI. This could be done in a backward compatible way by adding a keyword argument to the ``Extension`` class, such as:: Extension('foo', ['foo.c'], abi=3) Alternatives ============ In the initial python-dev thread [8]_ where this idea was first introduced, several alternatives were suggested. For completeness they are listed here, along with the reasons for not adopting them. Independent directories or symlinks ----------------------------------- Debian and Ubuntu could simply add a version-specific directory to ``sys.path`` that would contain just the extension modules for that version of Python. Or the symlink trick eliminated in PEP 3147 could be retained for just shared libraries. This approach is rejected because it propagates the essential complexity that PEP 3147 tries to avoid, and adds yet another directory to search for all modules, even when the number of extension modules is much fewer than the total number of Python packages. It also makes for more robust management when all of a package's module files live in the same directory, because it allows systems such as `dpkg` to detect file conflicts between distribution packages. Don't share packages with extension modules ------------------------------------------- It has been suggested that Python packages with extension modules not be shared among all supported Python versions on a distribution. Even with adoption of PEP 3149, extension modules will have to be compiled for every supported Python version, so perhaps sharing of such packages isn't useful anyway. Not sharing packages with extensions though is infeasible for several reasons. If a pure-Python package is shared in one version, should it suddenly be not-shared if the next release adds an extension module for speed? Also, even though all extension shared libraries will be compiled and distributed once for every supported Python, there's a big difference between duplicating the `.so` files and duplicating all `.py` files. The extra space increases the download time for such packages, and more immediately, increases the space pressures on already constrained distribution CD-ROMs. Reference implementation ======================== Work on this code is tracked in a Bazaar branch on Launchpad [5]_ until it's ready for merge into Python 3.2. The work-in-progress diff can also be viewed [6]_ and is updated automatically as new changes are uploaded. References ========== .. [1] PEP 3147 .. [2] PEP 384 .. [3] Ubuntu: http://www.ubuntu.com .. [4] Debian: http://www.debian.org .. [5] https://code.edge.launchpad.net/~barry/python/sovers .. [6] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411 .. [7] http://docs.python.org/py3k/distutils/index.html .. [8] http://mail.python.org/pipermail/python-dev/2010-June/100998.html Copyright ========= This document has been placed in the public domain.  .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

On Jul 16, 2010, at 12:16 AM, Amaury Forgeot d'Arc wrote:

2010/7/15 Barry Warsaw :

...

The first draft of PEP 3149 is ready for review.

I like it!

Cool!

I think it could mention the case where packages are not installed in the canonical directory, but placed elsewhere along the PYTHONPATH. This is how I deploy applications, for example, and the differences between python versions makes this a pain.

Because of the way the results of ./configure are embodied in the Makefile's $SO variable, and that variable is used unconditionally from sysconfig (and thus distutils), once you've configured --with-so-abi-tag, that tag will be used globally everywhere. I've added a note about this.

The specific case of Windows should be mentioned: each foo.pyd contains the name of the python library (Python27.dll) it has been linked with; It must be rebuilt for a major version change. IMO the Windows installers provided by python.org should be built with a tag that contains this major number.

The current version of the PEP and my implementation do not change the Windows builds at all. I don't feel qualified to integrate the ideas expressed in PEP 3149 for Windows builds, but I would be happy to accept patches to either the PEP or implementation to export the same tagging feature for Windows.

...

Thus for example, an initial implementation of PEP 384, compiled with `--with-so-abi-tag=cpython-xy` would search for the following file names when extension module `foo` is imported (in this order)::

   foo.abi3.so    foo.cpython-xy.so    foo.so

Is this the correct order? IMO the so-abi-tag is more precise and the two first items should be swapped.

Good point, fixed.

PyPy would also benefit from this patch: it can now use extension modules, but the ABI is slightly different. By default, PyPy would load only modules containing the ABI tag, and refuse foo.so which is incompatible for sure. But the installations could still be shared between Python implementations.

Interesting. I've added a note about this to the PEP. Thanks for the feedback. -Barry

On Fri, Jul 16, 2010 at 5:40 AM, Matthias Klose wrote:

2) As PEP 3147 defines a non-configurable name for .pyc files, this PEP should define a non-configurable way for the tag. The tag should include all information which currently makes an extension ABI incompatible: - the python implementation (cpython, PyPy, ...) - the python version (3,2, 3.3, ...) - unicode configure option (--with-wide-unicode, 16 or 32) - platform information (necessary?)

I'm not sure it's that easy to enumerate all of the ways to end up with an incompatible ABI. There are quite a lot of configure options listed with "configure --help", and it's not always obvious if an option changes the ABI. On top of that, there are compilation flags that can change the ABI, as Kristján discovered in the following thread: http://mail.python.org/pipermail/python-dev/2010-June/100583.html On the flip side, a fully enumerated ABI signature could be used to identify (in)compatible binary eggs, which is basically impossible now. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com

On Jul 16, 2010, at 12:40 PM, Matthias Klose wrote:

I like the proposal, but IMO it is too unspecific about the abi tag. Assume that an extension is built with such a configured python and then tried to run with an abi compatible configured python, but with a slightly different version tag, the extension won't be found. Differing file names per configuration should be avoided. Proposing

1) Remove the configure option and use the new naming using the tag for all configurations unconditionally. Everybody can expect the same name on every installation, and PEP 384 most likely will require using a tag too. As Amaury did point out, there is a use case in that PyPy can use this tag too to build extensions only usable by PyPy.

2) As PEP 3147 defines a non-configurable name for .pyc files, this PEP should define a non-configurable way for the tag. The tag should include all information which currently makes an extension ABI incompatible: - the python implementation (cpython, PyPy, ...) - the python version (3,2, 3.3, ...) - unicode configure option (--with-wide-unicode, 16 or 32) - platform information (necessary?) If this list changes for coming python versions, then it can be extended. Barry pointed out on irc chat that people might want to build experimental ABI incompatible versions, which should have its own tag. If this is wanted, please provide a configure option which lets extend/append to the tag.

Okay, I'm convinced we need to hard code this tag, and I think it's okay not to provide a configure switch to control this. I've pushed a new version of the diff and PEP, the latter attached here for your convenience. Thanks for all the feedback. -Barry PEP: 3149 Title: ABI version tagged .so files Version: $Revision: 81577 $ Last-Modified: $Date: 2010-05-27 19:54:25 -0400 (Thu, 27 May 2010) $ Author: Barry Warsaw Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2010-07-09 Python-Version: 3.2 Post-History: 2010-07-14, 2010-07-22 Resolution: TBD Abstract ======== PEP 3147 [1]_ described an extension to Python's import machinery that improved the sharing of Python source code, by allowing more than one byte compilation file (.pyc) to be co-located with each source file. This PEP defines an adjunct feature which allows the co-location of extension module files (.so) in a similar manner. This optional, build-time feature will enable downstream distributions of Python to more easily provide more than one Python major version at a time. Background ========== PEP 3147 defined the file system layout for a pure-Python package, where multiple versions of Python are available on the system. For example, where the `alpha` package containing source modules `one.py` and `two.py` exist on a system with Python 3.2 and 3.3, the post-byte compilation file system layout would be:: alpha/ __pycache__/ __init__.cpython-32.pyc __init__.cpython-33.pyc one.cpython-32.pyc one.cpython-33.pyc two.cpython-32.pyc two.cpython-33.pyc __init__.py one.py two.py For packages with extension modules, a similar differentiation is needed for the module's .so files. Extension modules compiled for different Python major versions are incompatible with each other due to changes in the ABI. Different configuration/compilation options for the same Python version can result in different ABIs (e.g. --with-wide-unicode). While PEP 384 [2]_ defines a stable ABI, it will minimize, but not eliminate extension module incompatibilities between Python builds or major versions. Thus a mechanism for discriminating extension module file names is proposed. Rationale ========= Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more than one Python version at the same time to their users. For example, Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, with Python 2.6 being the default. In order to share as much as possible between the available Python versions, these distributions install third party (i.e. non-standard library) packages into `/usr/share/pyshared` and symlink to them from `/usr/lib/pythonX.Y/dist-packages`. The symlinks exist because in a pre-PEP 3147 world (i.e < Python 3.2), the `.pyc` files resulting from byte compilation by the various installed Pythons will name collide with each other. For Python versions >= 3.2, all pure-Python packages can be shared, because the `.pyc` files will no longer cause file system naming conflicts. Eliminating these symlinks makes for a simpler, more robust Python distribution. A similar situation arises with shared library extensions. Because extension modules are typically named `foo.so` for a `foo` extension module, these would also name collide if `foo` was provided for more than one Python version. In addition, because different configuration/compilation options for the same Python version can cause different ABIs to be presented to extension modules. On POSIX systems for example, the configure options ``--with-pydebug``, ``--with-pymalloc``, and ``--with-wide-unicode`` all change the ABI. This PEP proposes to encode build-time options in the file name of the ``.so`` extension module files. PyPy [5]_ can also benefit from this PEP, allowing it to avoid name collisions in extension modules built for its API, but with a different `.so` tag. Proposal ======== The configure/compilation options chosen at Python interpreter build-time will be encoded in the shared library file name for extension modules. This "tag" will appear between the module base name and the operation file system extension for shared libraries. The following information *MUST* be included in the shared library file name: * The Python implementation (e.g. cpython, pypy, jython, etc.) * The interpreter's major and minor version numbers These two fields are separated by a hyphen and no dots are to appear between the major and minor version numbers. E.g. ``cpython-32``. Python implementations *MAY* include additional flags in the file name tag as appropriate. For example, on POSIX systems these flags will also contribute to the file name: * ``--with-pydebug`` (flag: ``d``) * ``--with-pymalloc`` (flag: ``m``) * ``--with-wide-unicode`` (flag: ``u``) By default in Python 3.2, ``configure`` enables ``--with-pymalloc`` so shared library file names would appear as ``foo.cpython-32m.so``. When the other two flags are also enabled, the file names would be ``foo.cpython-32dmu.so``. (This PEP only addresses build issues on POSIX systems that use the ``configure`` script. While Windows or other platform support is not explicitly disallowed under this PEP, platform expertise is needed in order to evaluate, describe, and implement support on such platforms.) The shared library file name tag is used unconditionally; it cannot be changed. The tag and extension module suffix are available through the ``sysconfig`` modules via the following variables:: >>> sysconfig.get_config_var('SO') '.cpython-32mu.so' >>> sysconfig.get_config_var('SOABI') 'cpython-32mu' Note that ``$SOABI`` contains just the tag, while ``$SO`` includes the platform extension for shared library files, and is the exact suffix added to the extension module name. For an arbitrary package `foo`, you would see these files when the distribution package was installed:: /usr/share/pyshared/foo.cpython-32m.so /usr/share/pyshared/foo.cpython-33m.so Python's dynamic module loader will recognize and import shared library extension modules with a tag that matches its build-time options. For backward compatibility, Python will also continue to import untagged extension modules, e.g. ``foo.so``. This shared library tag would be used globally for all distutils-based extension modules, regardless of where on the file system they are built. Extension modules built by means other than distutils would either have to calculate the tag manually, or fallback to the non-tagged `.so` file name. Proven approach =============== The approach described here is already proven, in a sense, on Debian and Ubuntu system where different extensions are used for debug builds of Python and extension modules. Debug builds on Windows also already use a different file extension for dynamic libraries, and in fact encoded (in a different way than proposed in this PEP) the Python major and minor version in the `.dll` file name. PEP 384 ======= PEP 384 defines a stable ABI for extension modules. In theory, universal adoption of PEP 384 would eliminate the need for this PEP because all extension modules could be compatible with any Python version. In practice of course, it will be impossible to achieve universal adoption, and as described above, different built-time flags still affect the ABI. Thus even with a stable ABI, this PEP may still be necessary. While a complete specification is reserved for PEP 384, here is a discussion of the relevant issues. PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is passed as the API version if the extension was complied with ``Py_LIMITED_API``. This should be formalized into an official macro called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``. If and when the ABI changes in an incompatible way, this version number would be bumped. To facilitate sharing, Python would be extended to search for extension modules with the ``PYTHON_ABI_VERSION`` number in its name. The prefix ``abi`` is reserved for Python's use. Thus, an initial implementation of PEP 384, when Python is configured with the default set of flags, would search for the following file names when extension module `foo` is imported (in this order):: foo.cpython-XYm.so foo.abi3.so foo.so The distutils [6]_ ``build_ext`` command would also have to be extended to compile to shared library files with the ``abi3`` tag, when the module author indicates that their extension supports that version of the ABI. This could be done in a backward compatible way by adding a keyword argument to the ``Extension`` class, such as:: Extension('foo', ['foo.c'], abi=3) Alternatives ============ In the initial python-dev thread [7]_ where this idea was first introduced, several alternatives were suggested. For completeness they are listed here, along with the reasons for not adopting them. Independent directories or symlinks ----------------------------------- Debian and Ubuntu could simply add a version-specific directory to ``sys.path`` that would contain just the extension modules for that version of Python. Or the symlink trick eliminated in PEP 3147 could be retained for just shared libraries. This approach is rejected because it propagates the essential complexity that PEP 3147 tries to avoid, and adds potentially several additional directories to search for all modules, even when the number of extension modules is much fewer than the total number of Python packages. For example, builds were made available both with and without wide unicode, with and without pydebug, and with and without pymalloc, the total number of directories search increases substantially. Don't share packages with extension modules ------------------------------------------- It has been suggested that Python packages with extension modules not be shared among all supported Python versions on a distribution. Even with adoption of PEP 3149, extension modules will have to be compiled for every supported Python version, so perhaps sharing of such packages isn't useful anyway. Not sharing packages with extensions though is infeasible for several reasons. If a pure-Python package is shared in one version, should it suddenly be not-shared if the next release adds an extension module for speed? Also, even though all extension shared libraries will be compiled and distributed once for every supported Python, there's a big difference between duplicating the `.so` files and duplicating all `.py` files. The extra size increases the download time for such packages, and more immediately, increases the space pressures on already constrained distribution CD-ROMs. Reference implementation ======================== Work on this code is tracked in a Bazaar branch on Launchpad [8]_ until it's ready for merge into Python 3.2. The work-in-progress diff can also be viewed [9]_ and is updated automatically as new changes are uploaded. References ========== .. [1] PEP 3147 .. [2] PEP 384 .. [3] Ubuntu: http://www.ubuntu.com .. [4] Debian: http://www.debian.org .. [5] http://codespeak.net/pypy/dist/pypy/doc/ .. [6] http://docs.python.org/py3k/distutils/index.html .. [7] http://mail.python.org/pipermail/python-dev/2010-June/100998.html .. [8] https://code.edge.launchpad.net/~barry/python/sovers .. [9] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411 Copyright ========= This document has been placed in the public domain.  .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

On Jul 22, 2010, at 03:58 PM, Ronald Oussoren wrote:

I guess this is not an explicit goal of this PEP, but the structure is very close to supporting multiple system architectures at the same time. I regularly develop code that needs to run on Windows, Linux and OSX and it is very convenient to do so in a shared directory tree (locally on one machine and accessed using remote mounts on the other ones). This works fine for pure python code, but I currently have to resort to tricks for extension modules.

[...]

A way to generically solve my problem is to add the platform name as well, such as "foo.cpython-32m-darwin.so" or "foo.cpython-32mu-linux2.so".

This could certainly be done in the Windows build, but that wouldn't help bridge the gap among different POSIX systems. I'd be open to adding the platform name to the tag, but I'd probably define it as part of the implementation field, e.g. foo.cpython-linux2-32m.so. Or maybe start with the platform name, e.g. foo.linux2-cpython-32m. This isn't a strong preference though. Thoughts? -Barry

On Jul 23, 2010, at 11:48 AM, Ronald Oussoren wrote:

...

I'd be open to adding the platform name to the tag, but I'd probably define it as part of the implementation field, e.g. foo.cpython-linux2-32m.so. Or maybe start with the platform name, e.g. foo.linux2-cpython-32m. This isn't a strong preference though.

I don't have a strong opionion, but placing the platform name at the start is probably better to be consistent with sysconfig.get_platform().

What about the architecture (i386, amd64)? With every increase in length I start to get more concerned. We could encode the platform and architecture, but that gets into cryptic territory. OTOH, would you really co-install i386 and amd64 shared libraries on the same machine? (hello NFS ;). -Barry

Ronald Oussoren writes:

On 23 Jul, 2010, at 11:54, Barry Warsaw wrote:

...

What about the architecture (i386, amd64)? With every increase in length I start to get more concerned. We could encode the platform and architecture, but that gets into cryptic territory. OTOH, would you really co-install i386 and amd64 shared libraries on the same machine? (hello NFS ;).

I don't need this, but then again I primarily use a platform where the vendor has a proper solution for having binaries for multiple architectures ;-)

Well, Apple doesn't prevent people from building 32/64 bit-only python installations. Doesn't that give you 3 choices i386, amd64, fat?? And you can have framework or non-framework builds. Doesn't anybody else think this is lost work for very little gain? My /usr/lib/python2.6/site-packages directory consumes 200MB on disk. I couldn't care less if my /usr/lib/python2.5/site-packages consumed the same amount of disk space... - Ralf

Barry Warsaw writes:

On Jul 23, 2010, at 01:46 PM, schmir@gmail.com wrote:

...

Doesn't anybody else think this is lost work for very little gain? My /usr/lib/python2.6/site-packages directory consumes 200MB on disk. I couldn't care less if my /usr/lib/python2.5/site-packages consumed the same amount of disk space...

Right, you probably don't care now that your extension modules live in foo.so so it probably won't make much difference if they were named foo-blahblah.so, as long as they import. :)

Most of the time it won't make much difference, right. But I can assure you, that it will bite some people and there is some code to be adapted.

If you use Debian or Ubuntu though, it'll be a win for you by allow us to make Python support much more robust.

I'd much prefer to have cleanly separated environments by having separate directories for my python modules. Sharing the source code and complicating things will not lead to increased robustness. - Ralf

Barry Warsaw writes:

Do you have concrete examples? Without that it's just speculation I can't do much to address. Is the problem big or small? Easy to work around or not?

Some of the things that need to be adapted are e.g. Makefiles (basically anything that assumes modules have a certain name), all of the freezers (cxFreeze, py2exe, ...). The biggest problem probably will be that an import will load the wrong module or no module at all. I'm just speculating here...

"Change is bad" isn't a constructive argument. ;)

Did I make that argument?

That's fine, but it's not the way Debian/Ubuntu works today. PEP 3149 adoption will definitely remove significant complication for deploying multiple Python versions at the same time on those systems.

You're just moving that complication into python. - Ralf

Matthias Klose writes:

Not true. Package managers like dpkg/apt-get, rpm/yum and maybe others do this for ages. And yes, the added "complexity" of package managers does lead to increased robustness.

but how does sharing things lead to increased robustness (even if it might be managed by your package manager)?

On Jul 26, 2010, at 10:53 PM, Ralf Schmitt wrote:

Some of the things that need to be adapted are e.g. Makefiles (basically anything that assumes modules have a certain name), all of the freezers (cxFreeze, py2exe, ...). The biggest problem probably will be that an import will load the wrong module or no module at all. I'm just speculating here...

I took a look at cx_freeze - it doesn't support Python 3.2 yet afaict (the build fails but it may be shallow). I'm going to look at py2exe as soon as I can get a Windows VM up and running. Since import is (usually) handled by the built-in dynload_shlib.c code it should generally Just Work I think, unless some application installs custom import hooks. In any case, I think this will be a fairly standard and probably simple, porting effort, which goes along with supporting any new major release.

...

"Change is bad" isn't a constructive argument. ;)

Did I make that argument?

Apologies.

...

That's fine, but it's not the way Debian/Ubuntu works today. PEP 3149 adoption will definitely remove significant complication for deploying multiple Python versions at the same time on those systems.

You're just moving that complication into python.

It's a much different level and scope of complexity though. For Python, it's pretty simple: look for an additional .so file name pattern which gets baked into Python at compile time. For that, you're able to remove a huge amount of complexity on Debian/Ubuntu by removing the need to manage multiple directory trees and symlinks between files in those trees. The tools that manage them (i.e. handle package installs and removals) also get much simpler. I think it's a worthwhile trade-off. Cheers, -Barry

On 23 Jul, 2010, at 23:19, Barry Warsaw wrote:

On Jul 23, 2010, at 12:54 PM, Barry Warsaw wrote:

...

On Jul 23, 2010, at 11:48 AM, Ronald Oussoren wrote:

...

...

I'd be open to adding the platform name to the tag, but I'd probably define it as part of the implementation field, e.g. foo.cpython-linux2-32m.so. Or maybe start with the platform name, e.g. foo.linux2-cpython-32m. This isn't a strong preference though.

I don't have a strong opionion, but placing the platform name at the start is probably better to be consistent with sysconfig.get_platform().

What about the architecture (i386, amd64)? With every increase in length I start to get more concerned. We could encode the platform and architecture, but that gets into cryptic territory. OTOH, would you really co-install i386 and amd64 shared libraries on the same machine? (hello NFS ;).

Thinking about this some more, I'd rather *not* include the platform or architecture in the tag by default. They aren't really necessary to support the instigating use case and will probably be fairly uncommon.

I'd be okay including a configure option to allow you to add whatever you want after the implementation, version, and flags. E.g. something like:

./configure --with-abi-tag-extension=linux2

would lead to foo.cpython-32m-linux2.so, so not the nicer names we'd prefer but probably good enough for your purposes.

Would that work for you?

That would certainly work. That said, I'm also fine with not adding the platform information or configure argument at all. My usecase is fairly exotic and I do have a feasible workaround. Ronald

On Fri, Jul 23, 2010 at 12:40 AM, Barry Warsaw wrote:

Python implementations *MAY* include additional flags in the file name tag as appropriate.  For example, on POSIX systems these flags will also contribute to the file name:

 * ``--with-pydebug`` (flag: ``d``)  * ``--with-pymalloc`` (flag: ``m``)  * ``--with-wide-unicode`` (flag: ``u``)

By default in Python 3.2, ``configure`` enables ``--with-pymalloc`` so shared library file names would appear as ``foo.cpython-32m.so``. When the other two flags are also enabled, the file names would be ``foo.cpython-32dmu.so``.

(This PEP only addresses build issues on POSIX systems that use the ``configure`` script.  While Windows or other platform support is not explicitly disallowed under this PEP, platform expertise is needed in order to evaluate, describe, and implement support on such platforms.)

This leads me to a question: how do these configure options affect the PEP 384 stable ABI? That PEP is currently silent on the issue, while PEP 3149 appears to implicitly assume that "abi3" completely specifies the ABI. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

...

This leads me to a question: how do these configure options affect the PEP 384 stable ABI? That PEP is currently silent on the issue, while PEP 3149 appears to implicitly assume that "abi3" completely specifies the ABI.

It's a great question - perhaps Martin can chime in? It may be that 'abiX' isn't enough to fully specify compatible extension modules even when that module is written entirely and solely against PEP 384. In that case, we may need to include the configure flags in the tag, e.g. foo.abi3-dmu.so.

The intention is that there is indeed just one stable ABI, so one configuration is the supported one, and that should be the "default" build. As for the specific settings, my analysis would be this: - pydebug: not supported by the stable ABI, as it changes the layout of PyObject, which is an exposed structure More specifically: Py_DEBUG, Py_TRACEREFS and Py_REF_DEBUG are all incompatible with the stable ABI - pymalloc: I fail to see the impact on the ABI. All allocator macros become function calls under Py_LIMITED_API, otherwise, there shouldn't be any need to have different versions of that. - wide-unicode: this is a tricky one. I'm tempted to say that the stable ABI should always use a Py_UNICODE that matches the platform's wchar_t. Alternative proposals are welcome. Regards, Martin

On 28 Aug, 2010, at 12:29, Martin v. Löwis wrote:

- wide-unicode: this is a tricky one. I'm tempted to say that the stable ABI should always use a Py_UNICODE that matches the platform's wchar_t. Alternative proposals are welcome.

Sizeof(wchar_t) is 4 on OSX, but the Apple frameworks use a 16-bit type to represent unicode codepoints (UniChar). Current builds on OSX use a 16-bit unicode type which makes it pretty cheap to convert strings from Python to a C array of UniChar. I'm therefore -1 on switching to a wide unicode build on OSX. Ronald

On Aug 31, 2010, at 11:22 AM, M.-A. Lemburg wrote:

BTW: Wasn't one of the main reasons for having versioned .so files the idea to be able to have UCS2 and UCS4 versions installed side-by-side ?

Yes. This isn't an issue for PEP 3149 because it adds a flag to the shared library file name for wide unicodes. It's an issue for PEP 384. -Barry

"Martin v. Löwis" wrote:

...

-1 on always using wchar_t as well. Python's default is UCS2 and the stable ABI should not change that.

It's not really Python's default. It is what configure.in does by default. Python's default, on Linux, is UCS-4.

No, the default is UCS2 on all platforms and in configure.in. configure.in only uses UCS4 if it finds a TCL installation that happens to use UCS4 - for some reason I don't know :-) However, most Linux distros and more recently also some BSDs have switched over to using UCS4 for their distribution versions of Python.

...

I also think that this information is not relevant for the stable ABI: Extensions that want to stick to the stable ABI should really not have to know whether Py_UNICODE maps to wchar_t or not. If they want to interface to a local whcar_t type they can use the conversion APIs we have for that in the Unicode API: PyUnicode_FromWideChar() and PyUnicode_AsWideChar().

Ok. I'm fine with excluding Py_UNICODE from the stable ABI. However, I think in the long run, I guess more support for wchar_t will then be needed in the API, e.g. more convenient argument parsing.

Sure, we could add that. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 07 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 7 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Amaury Forgeot d'Arc wrote:

Hi,

2010/9/7 M.-A. Lemburg :

...

...

Ok. I'm fine with excluding Py_UNICODE from the stable ABI. However, I think in the long run, I guess more support for wchar_t will then be needed in the API, e.g. more convenient argument parsing.

Sure, we could add that.

Just to be clear: does this mean that PyUnicode_FromUnicode() and PyUnicode_AsUnicode() won't belong to the stable ABI?

As I understood Martin's comment Py_UNICODE would not be part of the ABI in the sense that you can access the Py_UNICODE data from within the extension module. It should still be fine, passing around opaque Py_UNICODE buffers.

PyUnicode_AsWideChar() is not as fast, because it needs to copy the data.

True. Also see this patch which tries to address the issue: http://bugs.python.org/issue8654 With the terminology used there, the stable ABI would implicitly have Py_UNICODE_AGNOSTIC set - and then prevent exposing the structure of Py_UNICODE* buffers while still allowing to pass them around. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 07 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 7 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/