[Python-Dev] versioned .so files for Python 3.2

[Scott Dial]

On 6/26/2010 4:06 PM, Matthias Klose wrote:
> On 25.06.2010 22:12, James Y Knight wrote:
>> On Jun 25, 2010, at 4:53 AM, Scott Dial wrote:
>>> Placing .so files together does not simplify that install process in any
>>> way. You will still have to handle such packages in a special way.
>>
>> This is a good point, but I think still falls short of a solution. For a
>> package like lxml, indeed you are correct. Since debian needs to build
>> it once per version, it could just put the entire package (.py files and
>> .so files) into a different per-python-version directory.
> 
> This is what is currently done.  This will increase the size of packages
> by duplicating the .py files, or you have to install the .py in a common
> location (irrelevant to sys.path), and provide (sym)links to the
> expected location.

"This is what is currently done"  and "provide (sym)links to the
expected location" are conflicting statements. If you are symlinking .py
files from a shared location, then that is not the same as "just install
the package into a version-specific location". What motivation is there
for preferring symlinks?

Who cares if a ditro package install yields duplicate .py files? Nor am
I motivated by having to carry duplicate .py files in a distribution
package (I imagine the compression of duplicate .py files is amazing).

> A "different per-python-version directory" also has the disadvantage
> that file conflicts between (distribution) packages cannot be detected.

Why? That sounds like a broken tool, maybe I am naive, please explain.
If two packages install /usr/lib/python2.6/foo.so that should be just as
detectable two installing /usr/lib/python-shared/foo.cpython-26.so

If you *must* compile .so files for every supported version of python at
packaging time, then you are already saying the set of python versions
is known. I fail to see the difference between a package that installs
.py and .so files into many directories than having many .so files in a
single directory; except that many directories *already* works. The only
gain I can see is that you save duplicate .py files in the package and
on the filesystem, and I don't feel that gain alone warrants this
fundamental change.

I would appreciate a proper explanation of why/how a single directory is
better for your distribution. Also, I haven't heard anyone that wasn't
using debian tools chime in with support for any of this, so I would
like to know how this can help RPMs and ebuilds and the like.

> I don't think that installation into different locations based on the
> presence of extension will work.  Should a location really change if an
> extension is added as an optimization?  Splitting a (python) package
> into different installation locations should be avoided.

I'm not sure why changing paths would matter; any package that writes
data in its install location would be considered broken by your distro
already, so what harm is there in having the packaging tool move it
later? Your tool will remove the old path and place it in a new path.

All of these shenanigans seem to manifest from your distro's
python-support/-central design, which seems to be entirely motivated by
reducing duplicate files and *not* simplifying the packaging. While this
plan works rather well with .py files, the devil is in the details. I
don't think Python should be getting involved in what I believe is a
flawed design.

What happens to the distro packaging if a python package splits the
codebase between 2.x and 3.x (meaning they have distinct .py files)? As
someone else mentioned, how is virtualenv going to interact with
packages that install like this?

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu