request for comments - standardization of python's purelib and platlib
Hello, I'm cross-posting this to distributions@freedesktop and python-dev, because the topic is relevant to both groups and should be solved in cooperation. The issue: In Python's default configuration (on linux), both purelib (location for pure python modules) and platlib (location for platform-dependent binary extensions) point to $prefix/lib/pythonX.Y/site-packages. That is no good for two main reasons. One, python depends on the "lib" directory. (from distro's point of view, prefix is /usr, so let's talk /usr/lib) Due to this, it's impossible to install python under /usr/lib64 without heavy patching. Repeated attempts to bring python developers to acknowledge and rectify the situation have all failed (common argument here is "that would mean redesign of distutils and huge parts of whatnot"). Conversely, that also means that multiarch setup (/usr/lib or lib32 with 32bit python and /usr/lib64 with 64bit python) is not possible with stock python. Two, the default configuration makes purelib and platlib identical, which somehow defeats the purpose of the distinction in the first place. You either need to patch the default, or supply some alternate configuration to take advantage of this feature. And that's not the end of it - the next step is to make python aware of two different locations on sys.path, one for purelib and one for platlib, which is a different story altogether. As distributors, we like to take advantage of purelib/platlib separation to package pure python modules as platform-independent (noarch for rpm-speakers). And that's not easy to do properly. The proposal: Let's put our heads together and choose good default locations for purelib and platlib. Then add support to python for recognizing the locations by default, and possibly leave note in FHS that "this is the place". This is IMO a good first step to making python multiarch-aware, and it would also help a bit with LSB integration [1]. I've come up with three basic options for the configuration (substitute "/usr" with "$prefix" if you're not a distributor). This list is by no means comprehensive, it's just what looked reasonable at the time of writing. 1 - the traditional way purelib = /usr/lib/pythonX.Y/site-packages platlib = /usr/lib(64)/pythonX.Y/site-packages pros: + this is already the default for 32bit systems + major distributions (including Fedora, Mandriva and now finally openSUSE too) do this cons: - 32bit systems have no separation, poor they! - with multiarch setup, /usr/lib is "cluttered" by both platform-dependent files for 32bit and platform-independent files shared by the platforms. Also, 64bit python can pick up 32bit modules. That doesn't cause problems in practice, but doesn't fell like a clean design. 2 - the sharedir way purelib = /usr/share/python/X.Y platlib = /usr/lib(64)/pythonX.Y/site-packages pros: + clean separation of purelib - nice! + unheard of - a good place to start anew cons: - FHS states that /usr/share is for data. But OTOH, they don't say much about platform-independent bytecode. We could probably get an exception for this. - unheard of - everyone will be surprised 3 - the perl way purelib = /usr/lib/pythonX.Y platlib = /usr/lib/pythonX.Y/lib-dynload-(platform-identifier)/site-packages pros: + possibility of multiarch packages that would install pure python parts into purelib and extensions or accelerators for more platforms at once - and therefore, possibility to split large modules into platform-dependent and platform-independent parts and save space on installation media + "idea compatibility" with perl and ruby, one less install layout to learn cons: - completely different from what we have now - would require the most work from both python developers and distributions comments? regards jan matejek python packager for SUSE Linux [1] http://www.linuxfoundation.org/en/LsbPython
On Thu, Aug 13, 2009 at 11:23, Jan Matejek <jan.matejek@novell.com> wrote:
This is now Tarek's call, so this may or may not have changed in terms of what the (now) distutils maintainer thinks.
Why can't pure libraries go into lib64 as well? There is nothing saying that a pure Python package won't have a setup.py that installs different files based on whether it is for a 32-bit or 64-bit CPython install.
Now are you proposing that packages that have both Python source and extensions be split based on the type of files, or that only pure Python packages go to /usr/share/python and any packages that are mixed go into lib(64)? If you are proposing the latter this is more reasonable as the former will require using .pth files to get import to search both locations for files in the same package and that just feels icky to me.
I think that last con says what chances this approach has of winning. =) -Brett
On Thu, Aug 13, 2009 at 9:22 PM, Brett Cannon<brett@python.org> wrote:
I don't recall those repeated attempts , but I've been around for less than two years. You are very welcome to come in the Distutils-SIG ML to discuss these matters. I'm moving the discussion there. Among the proposals you have detailed, the sharedir way seems like the most simple/interesting one (depending on you answer to Brett's question ) Regards Tarek -- Tarek Ziadé | http://ziade.org
Hi Tarek, What is needed is to remove/refactor the hardcoding of paths that currently exists within distutils and replace it with the ability to override the defaults via configuration files. (distutils.cfg?) If there's one thing that's certain for the future, its that python will go onto more platforms. Using different paths. When people are complaining about paths being hard-coded into distutils and it causing angst, I think that their complaints are valid. I can find posts going back to 2004 for windows users complaining about exactly the same thing. So it isn't a new issue. The problem applies to both linux and windows. Anyway.. do you know the code that we're talking about? David On Fri, 14 Aug 2009 10:02:03 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On 14.08.2009 10:02, Tarek Ziadé wrote:
The approach of splitting the installation into two different locations seems to be wrong, it changes the semantics for imports of python packages which are not installed in the same location. Simplest counter example is the use of relative imports, which will fail if the imported module/extension is not found in the same paths. Other languages like Perl or Java don't have relative imports, or they map all components on the "path" into one logical path so you don't have this kind of problem. I don't see an explict statement that code really has to live inside /usr/share, and even generated .py files differ depending on the architecture you build for (e.g. sip, qt bindings). Matthias
Dne 13.8.2009 21:22, Brett Cannon napsal(a):
What i'd like to accomplish is to have pure "noarch" package that can be installed unchanged into 32bit or 64bit (or 256bit) system, and the respective python would still find the files. Or, to put it another way, a package that can be installed into a multiarch system and be recognized by pythons of all architectures (assuming they are the same version, of course). If the distutils package installs different pure files for 32bit and 64bit python, then it can't be "noarch", so it doesn't matter if it goes into lib64. Also, such package would break this particular scheme - in the situation where the user installs only 32bit version of such package and tries to run it with 64bit python, it will probably break in some weird way. Last but not least, i'd argue that if a python-only package installs different files for different platforms, it is platform-dependent and therefore not pure ;)
The latter. Assume no change to "normal" distutils mechanism, only setting the default paths. (for now anyway) regards m.
Hi Jan, It's not impossible, but you have some dependencies. If you can patch distutils within Suse, then it mightn't be so difficult. Distutils is not much more than a file copier. Inside distutils, a lot of the paths that you are talking about are hardcoded.
correction - light patching.
Make it a zope/plone issue... and something might get done about it.... haha If it's a windows or linux issue... pour petrol on it and light a match.. seriously... it's not major refactoring.. it's just changing a few conditional constants.. within distutils..
Sure - discuss away. But you might end up having to patch your own distribution.
+1 Go try... David http://sourceforge.net/projects/pythonpkgmgr/
On Thu, Aug 13, 2009 at 11:23, Jan Matejek <jan.matejek@novell.com> wrote:
This is now Tarek's call, so this may or may not have changed in terms of what the (now) distutils maintainer thinks.
Why can't pure libraries go into lib64 as well? There is nothing saying that a pure Python package won't have a setup.py that installs different files based on whether it is for a 32-bit or 64-bit CPython install.
Now are you proposing that packages that have both Python source and extensions be split based on the type of files, or that only pure Python packages go to /usr/share/python and any packages that are mixed go into lib(64)? If you are proposing the latter this is more reasonable as the former will require using .pth files to get import to search both locations for files in the same package and that just feels icky to me.
I think that last con says what chances this approach has of winning. =) -Brett
On Thu, Aug 13, 2009 at 9:22 PM, Brett Cannon<brett@python.org> wrote:
I don't recall those repeated attempts , but I've been around for less than two years. You are very welcome to come in the Distutils-SIG ML to discuss these matters. I'm moving the discussion there. Among the proposals you have detailed, the sharedir way seems like the most simple/interesting one (depending on you answer to Brett's question ) Regards Tarek -- Tarek Ziadé | http://ziade.org
Hi Tarek, What is needed is to remove/refactor the hardcoding of paths that currently exists within distutils and replace it with the ability to override the defaults via configuration files. (distutils.cfg?) If there's one thing that's certain for the future, its that python will go onto more platforms. Using different paths. When people are complaining about paths being hard-coded into distutils and it causing angst, I think that their complaints are valid. I can find posts going back to 2004 for windows users complaining about exactly the same thing. So it isn't a new issue. The problem applies to both linux and windows. Anyway.. do you know the code that we're talking about? David On Fri, 14 Aug 2009 10:02:03 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On 14.08.2009 10:02, Tarek Ziadé wrote:
The approach of splitting the installation into two different locations seems to be wrong, it changes the semantics for imports of python packages which are not installed in the same location. Simplest counter example is the use of relative imports, which will fail if the imported module/extension is not found in the same paths. Other languages like Perl or Java don't have relative imports, or they map all components on the "path" into one logical path so you don't have this kind of problem. I don't see an explict statement that code really has to live inside /usr/share, and even generated .py files differ depending on the architecture you build for (e.g. sip, qt bindings). Matthias
Dne 13.8.2009 21:22, Brett Cannon napsal(a):
What i'd like to accomplish is to have pure "noarch" package that can be installed unchanged into 32bit or 64bit (or 256bit) system, and the respective python would still find the files. Or, to put it another way, a package that can be installed into a multiarch system and be recognized by pythons of all architectures (assuming they are the same version, of course). If the distutils package installs different pure files for 32bit and 64bit python, then it can't be "noarch", so it doesn't matter if it goes into lib64. Also, such package would break this particular scheme - in the situation where the user installs only 32bit version of such package and tries to run it with 64bit python, it will probably break in some weird way. Last but not least, i'd argue that if a python-only package installs different files for different platforms, it is platform-dependent and therefore not pure ;)
The latter. Assume no change to "normal" distutils mechanism, only setting the default paths. (for now anyway) regards m.
Hi Jan, It's not impossible, but you have some dependencies. If you can patch distutils within Suse, then it mightn't be so difficult. Distutils is not much more than a file copier. Inside distutils, a lot of the paths that you are talking about are hardcoded.
correction - light patching.
Make it a zope/plone issue... and something might get done about it.... haha If it's a windows or linux issue... pour petrol on it and light a match.. seriously... it's not major refactoring.. it's just changing a few conditional constants.. within distutils..
Sure - discuss away. But you might end up having to patch your own distribution.
+1 Go try... David http://sourceforge.net/projects/pythonpkgmgr/
participants (5)
-
Brett Cannon -
David Lyon -
Jan Matejek -
Matthias Klose -
Tarek Ziadé