[Distutils] Handling the binary dependency management problem

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/02/2013 12:23 PM, Vinay Sajip wrote:
> On Mon, 2/12/13, Tres Seaver <tseaver at palladion.com> wrote:
> 
>> The issue is combinatorial explosion in the compatibility  tag
>> space. There is basically zero chance that even Linux users (even
>> RedHat users  across RHEL version) would benefit from pre-built
>> binary wheels (as  opposed to packages from their distribution).
>> Wheels on POSIX allow caching of the build process for deployment
>> across a known set of hosts: they won't insulate you from the need
>> to build in the first place.
> 

> The combinations are number of Python X.Y versions x the no. of
> platform architectures/ABI variants, or do you mean something more
> than this?

Trying to mark up wheels so that they can be safely shared with unknown
POSIXy systems seems like a halting problem, to me:  the chance I can
build a wheel on my machine that you can use on yours (the only reason to
distribute a wheel, rather than the sdist, in the first place) drops off
sharply as wheel's "binariness" comes into play.  I'm arguing that wheel
is not an interesting *distribution* format for POSIX systems (at least,
for non-Mac ones).  It could still play out in *deployment* scenarios (as
you note below).

Note that wheel's main deployment advantage over a binary egg
(installable by pip) is exactly reversed if you use 'easy_install' or
'zc.buildout'.  Otherwise, in a controlled deployment, they are pretty
much equivalent.

> The wheel format is supposed to be a cross-platform binary package 
> format; are you saying it is completely useless for POSIX except as a 
> cache for identical hosts? What about for the cases like simple C 
> extensions which have no external dependencies, but are only for 
> speedups?

I have a lot of packages on PyPI which have such optimization-only
speeedups.  The time difference to build such extensions is trivial
(e.g., for zope.interface, ~1 second on my old slow laptop, versus 0.4
seconds without the extension).

Even for lxml (Daniel's original motivating case), the difference is ~45
seconds to build from source vs. 1 second to install a wheel (or and
egg).  The instant I have to think about whether the binary form might be
subtly incompatbile, that 1 second *loses* to the 45 seconds I spend over
here arguing with you guys while it builds again from source. :)

> What about POSIX environments where compilers aren't available (e.g.
> restricted/embedded environments, or due to security policies)?

Such environments are almost certainly driven by development teams who
can build wheels specifically for deployment to them (assuming the
policies allow anything other than distro-package-managed software).
This is still really a "cache the build" optimization to known platforms
(w/ all binary dependencies the same), rather than distribution.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKcyPsACgkQ+gerLs4ltQ4oBwCgvhoq8ovEn/Bl/0FpBEfI48JY
znEAoJElD+R9SPnJXduwjCy7oxWRmcWH
=a0TT
-----END PGP SIGNATURE-----