[Python-Dev] request for comments - standardization of python's purelib and platlib

[Brett Cannon]

On Thu, Aug 13, 2009 at 11:23, Jan Matejek <jan.matejek at novell.com> wrote:

> Hello,
>
> I'm cross-posting this to distributions at freedesktop and python-dev,
> because the topic is relevant to both groups and should be solved in
> cooperation.
>
> The issue:
>
> In Python's default configuration (on linux), both purelib (location for
> pure python modules) and platlib (location for platform-dependent binary
> extensions) point to $prefix/lib/pythonX.Y/site-packages.
> That is no good for two main reasons.
>
> One, python depends on the "lib" directory. (from distro's point of
> view, prefix is /usr, so let's talk /usr/lib) Due to this, it's
> impossible to install python under /usr/lib64 without heavy patching.
> Repeated attempts to bring python developers to acknowledge and rectify
> the situation have all failed (common argument here is "that would mean
> redesign of distutils and huge parts of whatnot").
>

This is now Tarek's call, so this may or may not have changed in terms of
what the (now) distutils maintainer thinks.


>
> Conversely, that also means that multiarch setup (/usr/lib or lib32 with
> 32bit python and /usr/lib64 with 64bit python) is not possible with
> stock python.
>
> Two, the default configuration makes purelib and platlib identical,
> which somehow defeats the purpose of the distinction in the first place.
> You either need to patch the default, or supply some alternate
> configuration to take advantage of this feature.
> And that's not the end of it - the next step is to make python aware of
> two different locations on sys.path, one for purelib and one for
> platlib, which is a different story altogether.
>
> As distributors, we like to take advantage of purelib/platlib separation
> to package pure python modules as platform-independent (noarch for
> rpm-speakers). And that's not easy to do properly.
>
> The proposal:
>
> Let's put our heads together and choose good default locations for
> purelib and platlib. Then add support to python for recognizing the
> locations by default, and possibly leave note in FHS that "this is the
> place".
>
> This is IMO a good first step to making python multiarch-aware, and it
> would also help a bit with LSB integration [1].
>
> I've come up with three basic options for the configuration (substitute
> "/usr" with "$prefix" if you're not a distributor). This list is by no
> means comprehensive, it's just what looked reasonable at the time of
> writing.
>
> 1 - the traditional way
> purelib = /usr/lib/pythonX.Y/site-packages
> platlib = /usr/lib(64)/pythonX.Y/site-packages
>

Why can't pure libraries go into lib64 as well? There is nothing saying that
a pure Python package won't have a setup.py that installs different files
based on whether it is for a 32-bit or 64-bit CPython install.


>
> pros:
> + this is already the default for 32bit systems
> + major distributions (including Fedora, Mandriva and now finally
> openSUSE too) do this
> cons:
> - 32bit systems have no separation, poor they!
> - with multiarch setup, /usr/lib is "cluttered" by both
> platform-dependent files for 32bit and platform-independent files shared
> by the platforms. Also, 64bit python can pick up 32bit modules. That
> doesn't cause problems in practice, but doesn't fell like a clean design.
>
> 2 - the sharedir way
> purelib = /usr/share/python/X.Y
> platlib = /usr/lib(64)/pythonX.Y/site-packages


Now are you proposing that packages that have both Python source and
extensions be split based on the type of files, or that only pure Python
packages go to /usr/share/python and any packages that are mixed go into
lib(64)? If you are proposing the latter this is more reasonable as the
former will require using .pth files to get import to search both locations
for files in the same package and that just feels icky to me.


>
> pros:
> + clean separation of purelib - nice!
> + unheard of - a good place to start anew
> cons:
> - FHS states that /usr/share is for data. But OTOH, they don't say much
> about platform-independent bytecode. We could probably get an exception
> for this.
> - unheard of - everyone will be surprised


> 3 - the perl way
> purelib = /usr/lib/pythonX.Y
> platlib =
> /usr/lib/pythonX.Y/lib-dynload-(platform-identifier)/site-packages
>
> pros:
> + possibility of multiarch packages that would install pure python parts
> into purelib and extensions or accelerators for more platforms at once -
> and therefore, possibility to split large modules into
> platform-dependent and platform-independent parts and save space on
> installation media
> + "idea compatibility" with perl and ruby, one less install layout to learn
> cons:
> - completely different from what we have now - would require the most
> work from both python developers and distributions
>

I think that last con says what chances this approach has of winning. =)

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090813/fb45ec14/attachment-0001.htm>