Re: [Distutils] How to make easy_install handle platlibs?
At 08:53 PM 4/10/2009 -0700, Buck wrote:
I see the kernel version and architecture, but this is insufficient; RedHat 4 and RedHat 5 both use a 2.6 kernel, but the difference in provided libraries are sufficient to make many (most?) "impure" libraries stop working ( numpy, python-ldap, and hashlib for examples).
easy_install apparently knows when packages are platform-dependant, and the necessary directory is sys.exec_prefix (or maybe better: distutils.sysconfig.get_python_lib(plat_specific=1) ), so this seems like an easy change. Would a patch be accepted?
It would be a pretty major patch, since the distutils logic that's used for determining the installation directory is applied long before easy_install even knows what it's installing (and therefore, whether it's platform dependent). It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X.
On Fri, Apr 10, 2009 at 9:15 PM, P.J. Eby <pje@telecommunity.com> wrote:
At 08:53 PM 4/10/2009 -0700, Buck wrote:
I see the kernel version and architecture, but this is insufficient; RedHat 4 and RedHat 5 both use a 2.6 kernel, but the difference in provided libraries are sufficient to make many (most?) "impure" libraries stop working ( numpy, python-ldap, and hashlib for examples).
easy_install apparently knows when packages are platform-dependant, and the necessary directory is sys.exec_prefix (or maybe better: distutils.sysconfig.get_python_lib(plat_specific=1) ), so this seems like an easy change. Would a patch be accepted?
It would be a pretty major patch, since the distutils logic that's used for determining the installation directory is applied long before easy_install even knows what it's installing (and therefore, whether it's platform dependent).
It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X.
If the distutils logic is used, then why does numpy install to the platlib folder using distutils, but into the nonplatlib folder using easy_install?
Your tone seems to suggest that such an effort might be accepted upstream. --Buck
It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X.
As PJE has mentioned, the intent is that the egg name should contain enough information to decide if that egg will work on your platform. For example, if it says "py2.5-win32" then you know it will work on your Python 2.5 on 32-bit Windows, and if it says "py2.5-macosx-10.3- ppc" then you know it will work on your PowerPC-based Mac with Python 2.5. This can be used for example by easy_install to get a directory listing of eggs and choose which one will work on the local platform based solely on its filename. As PJE mentioned, it would be nice if this same technique worked on Linux. However, it currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. To know whether such an egg would actually work on your Linux system, you would also need to know whether the Python was compiled with UCS-2 or UCS-4 internal unicode representation, as well as what version of glibc you have. Is there anything else that would need to be added into the egg name? Regards, Zooko
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 zooko wrote:
It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X.
As PJE has mentioned, the intent is that the egg name should contain enough information to decide if that egg will work on your platform. For example, if it says "py2.5-win32" then you know it will work on your Python 2.5 on 32-bit Windows, and if it says "py2.5-macosx-10.3- ppc" then you know it will work on your PowerPC-based Mac with Python 2.5. This can be used for example by easy_install to get a directory listing of eggs and choose which one will work on the local platform based solely on its filename.
As PJE mentioned, it would be nice if this same technique worked on Linux.
However, it currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. To know whether such an egg would actually work on your Linux system, you would also need to know whether the Python was compiled with UCS-2 or UCS-4 internal unicode representation, as well as what version of glibc you have. Is there anything else that would need to be added into the egg name?
Better: just don't distribute binary eggs for Linux at all: - Developers have the tools (or can install them) to build from 'sdists'. - Systems without tools are "locked down", which means that they shouldn't be installing from public distributions anyway: the folks who run them should be able to build *private* eggs from 'sdists' which are known to work for their systems. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ4g3k+gerLs4ltQ4RAjepAKDK9/Y7Gq218CtUOw2KRtAkbLYcWACgtYF5 ogSEjrYyCWzliUo5/jHGm94= =4smt -----END PGP SIGNATURE-----
zooko wrote:
However, it currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. To know whether such an egg would actually work on your Linux system, you would also need to know whether the Python was compiled with UCS-2 or UCS-4 internal unicode representation, as well as what version of glibc you have. Is there anything else that would need to be added into the egg name?
Yes, if you used symbols from any shared library in an extension module, you'd need to know the version of that shared library. So it's not just libc. This is the same on any OS, not just linux. One example is anything that uses the numpy C API (e.g. matplotlib, pyopengl 2.x, and so on). Fortunately, the numpy C API is very stable, so there hasn't been any incompatibility introduced in the numpy 1.x series. Yet. -Andrew
On Sun, Apr 12, 2009 at 8:54 AM, Andrew Straw <strawman@astraw.com> wrote:
zooko wrote:
However, it currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. To know whether such an egg would actually work on your Linux system, you would also need to know whether the Python was compiled with UCS-2 or UCS-4 internal unicode representation, as well as what version of glibc you have. Is there anything else that would need to be added into the egg name?
Yes, if you used symbols from any shared library in an extension module, you'd need to know the version of that shared library. So it's not just libc. This is the same on any OS, not just linux.
-Andrew
Exactly. The platform name/version ( RedHat 3 / Ubuntu 7.07 / Gentoo X.X ) can serve as a much better proxy for the "version of all shared libraries" than just the kernel version alone (2.4 / 2.6). To add to Andrew's example, python-ldap created for Redhat 4 depends on ldap_r.so.1.0.4 but for RedHat 5 depends on ldap_r.so.1.0.6. Both have kernel 2.6, so are indistinguishable under current naming scheme. I notice that for Debian/Ubuntu the lsb_release command is implemented in Python, although it's mostly just reading out /etc/debian_version. RedHat's lsb_release uses bash, but similarly just reads in a file. Gentoo also has a lsb_release. The "completely correct" way to do this would be to run ldd on all binaries, stick the result in a metadata file somewhere, then load the version of the egg where the most (hopefully all) libraries exist. On the other hand, sticking the egg into the place that distutils uses when not under easy_install would fix this much more simply, although from what I hear this would be a big change. -- Buck Golemon
Yes, if you used symbols from any shared library in an extension module, you'd need to know the version of that shared library. So it's not just libc. This is the same on any OS, not just linux.
Wait a minute, an extension module built into the Python Standard Library, you mean? Because for separately packaged packages ("distributions") such as the numpy that you mentioned, your package ("distribution") would express its requirement on that other package ("distribution") in its install_requires metadata, not in its name. There, in the install_requires metadata, it can also express which version it requires. Right? Regards, Zooko
zooko wrote:
Yes, if you used symbols from any shared library in an extension module, you'd need to know the version of that shared library. So it's not just libc. This is the same on any OS, not just linux.
Wait a minute, an extension module built into the Python Standard Library, you mean?
No, I'm writing about non stdlib extension modules. Because for separately packaged packages
("distributions") such as the numpy that you mentioned, your package ("distribution") would express its requirement on that other package ("distribution") in its install_requires metadata, not in its name. There, in the install_requires metadata, it can also express which version it requires. Right?
No, because the act of compiling your .egg fixes the specific version (e.g. numpy==1.3) to keep the ABI compatible with the version of numpy installed at extension module compile time. Whereas the install_requires is about API compatibility, and could thus be numpy>=1.2, for example. (For pure Python modules, this isn't a problem because there is only an API version to worry about. But with compiled extensions, there is also the ABI version to worry about.)
On Apr 12, 8:51 am, Tres Seaver <tsea...@palladion.com> wrote:
zooko wrote:
It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X.
However, it (egg naming scheme on Linux) currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. ...
Better: just don't distribute binary eggs for Linux at all:
- Developers have the tools (or can install them) to build from 'sdists'.
- Systems without tools are "locked down", which means that they shouldn't be installing from public distributions anyway: the folks who run them should be able to build *private* eggs from 'sdists' which are known to work for their systems.
Tres.
I have no clue what you mean by 'sdists'. Is this a widely-known thing? A URL to an example would be sufficient. Thanks, --Buck
David Cournapeau <david@ar.media.kyoto-u.ac.jp> writes:
Buck wrote:
I have no clue what you mean by 'sdists'. Is this a widely-known thing?
sdist is the name of the distutils command - it just means distributing a source tarball.
Again, please stop blurring the distinction. I can only assume at this point that it's deliberate on your part, but I have no idea why. A distutils ‘sdist’ is a combination of the source files *and* metadata for distutils. A mere source tarball is *not* an sdist. -- \ “I know you believe you understood what you think I said, but I | `\ am not sure you realize that what you heard is not what I | _o__) meant.” —Robert J. McCloskey | Ben Finney
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Cournapeau wrote:
Buck wrote:
I have no clue what you mean by 'sdists'. Is this a widely-known thing?
sdist is the name of the distutils command - it just means distributing a source tarball.
Nope, by "sdist" I mean the specific tarball generated by the 'sdist' command (PJE spelled out what is special about them). Manually-created source tarballs don't have the layout / metadata which makes automating their installation feasible. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ5h3w+gerLs4ltQ4RAjMUAJ0bG4aCjLVeC7J2czCbrcMQshxg1QCfexDG u0/T9gwas+IJeqN+4GsFcsM= =l29V -----END PGP SIGNATURE-----
participants (8)
-
Andrew Straw -
Ben Finney -
Buck -
Buck Golemon
-
David Cournapeau -
P.J. Eby -
Tres Seaver -
zooko