Hi, On 17 July 2015 at 05:22, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 17 July 2015 at 03:41, Nate Coraor <nate@bx.psu.edu> wrote:
[...]
As mentioned in the wheels PR, there are some questions and decisions made that I need guidance on:
- On Linux, the distro name/version (as determined by platform.linux_distribution()) will be appended to the platform string, e.g. linux_x86_64_ubuntu_14_04. This is going to be necessary to make a reasonable attempt at wheel compatibility in PyPI. But this may violate PEP 425.
I think it's going beyond it in a useful way, though. At the moment, the "linux_x86_64" platform tag *under*specifies the platform - a binary extension built on Ubuntu 14.04 with default settings may not work on CentOS 7, for example.
Adding in the precise distro name and version number changes that to *over*specification, but I now think we can address that through configuration settings on the installer side that allow the specification of "compatible platforms". That way a derived distribution could add the corresponding upstream distribution's platform tag and their users would be able to install the relevant wheel files by default. [...]
The definition of "acceptable platform tags" should list the platforms in order of preference (for example, some of the backward compatible past releases of a linux distro, in reverse order), so that if multiple acceptable wheels are present the closest one is selected. As some other have mentioned, this doesn't solve the problem of system dependencies. I.e.: a perfectly compiled lxml wheel for linux_x86_64_ubuntu_14_04, installed into Ubuntu 14.04, will still fail to work if libxml2 and libxslt1.1 debian packages are not installed (among others). Worse is that pip will gladly install such package, and the failure will happen as a potentially cryptic error message payload to an ImportError that doesn't really make it clear what needs to be done to make the package actually work. To solve this problem, so far we've only been able to come up with two extremes: - Have the libraries contain enough metadata in their source form that we can generate true system packages from them (this doesn't really help the virtualenv case) - Carry all the dependencies. Either by static linking, or by including all dynamic libraries in the wheel, or by becoming something like Conda where we package even non Python projects. As a further step that could be taken on top of Nate's proposed PR, but avoiding the extremes above, I like Daniel's idea of "specifying the full library names [...] à-lá RPM". Combine it with the specification of abstract locations, and we could have wheels declare something like. - lxml wheel for linux_x86_64_ubuntu_14_04: - extdeps: - <dynlibdir>/libc.so.6 - <dynlibdir>/libm.so.6 - <dynlibdir>/libxml2.so.2 - <dynlibdir>/libexslt.so.0 This also makes it possible to have wheels depend on stuff other than libraries, for example binaries or data files (imagine a lightweight version of pytz that didn't have to carry its own timezones, and depended on the host system to keep them updated). As long as we have a proper abstract location to anchor the files, we can express these dependencies without hardcoding paths as they were on the build machine. It even opens the possibility that some of these external dependencies could be provided on a per-virtualenv basis, instead of globally. Pip could then (optionally?) check the existence of these external dependencies before allowing installation of the wheel, increasing the likelihood that it will work once installed. This same way of expressing external dependencies could be extended to source packages themselves. For example the `setup()` (or whatever successor we end up with) for a PIL source package could express dependency on '<include>/png.h'. Or, what's more likely these days, a dependency on '<bindir>/libpng12-config', which when run prints the correct invocations of gcc flags to add to the build process. The build process would then check the presence of these external build dependencies early on, allowing for much clearer error messages and precise instructions on how to provide the proper build environment. Most distros provide handy ways of querying which packages provide which files, so I believe the specification of external file dependences to be a nice step up from where we are right now, without wading into full-system-integration territory. Leo