[Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker <chris.barker at noaa.gov>
wrote:

> NOTE: I don't work for any of the companies involved -- just a somewhat
> frustrated user... And someone that has been trying for years to make
> things easier for OS-X users.
>
> I’m not sure what (3) means exactly. What is a “normal” Python, do you
>>> modify Python in a way that breaks the ABI but which isn’t reflected in the
>>> standard ABI tag?
>>>
>>
>> It could be multiple things. The most obvious one is that generally.
>> cross-platforms python distributions will try to be "relocatable" (i.e. the
>> whole installation can be moved and still work). This means they require
>> python itself to be built a special way. Strictly speaking, it is not an
>> ABI issue, but the result is the same though: you can't use libraries from
>> anaconda or canopy on top of a normal python
>>
>
> But why not? -- at least for Anaconda, it's because those libraries likely
> have non-python dependencies, which are expected to be installed in a
> particular way. And really, this is not particular to Anaconda/Canopy at
> all. Python itself has no answer for this issue, and eggs and wheels don't
> help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order
> to build a binary wheel with non-python dependencies (let's say something
> like libjpeg, for instance), you need to either:
>  - assume that libjpeg is installed in a "standard" place -- really no
> solution at all (at least outside of linux)
>  - statically link it
>  - ship the dynamic lib with the package
>
> For the most part, the accepted solution for OS-X has been to statically
> link, but:
>
>  - it's a pain to do. The gnu toolchain really likes to use dynamic
> linking, and building a static lib that will run on a
> maybe-older-than-the-build-system machine is pretty tricky.
>
>  - now we end up with multiple copies of the same lib in the python
> install. There are a handful of libs that are used a LOT. Maybe there is no
> real downside -- disk space and memory are cheap these days, but it sure
> feels ugly. And I have yet to feel comfortable with having multiple
> versions of the same lib linked into one python instance -- I can't say
> I've seen a problem, but it makes me nervous.
>
> On Windows, the choices are the same, except that: It is so much harder to
> build many of the "standard" open source libs that package authors are more
> likely to do it for folks, and you do get the occasional "dll hell" issues.
>
> I had a plan to make some binary wheels for OS-X that were not really
> python packages, but actually just bundled up libs, so that other wheels
> could depend on them. OS-X does allow linking to relative paths, so this
> should have been doable, but I never got anyone else to agree this was a
> good idea, and I never found the roundtoits anyway. And it doesn't really
> fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are
> platform dependent and even worse, build-dependent.
>
> Meanwhile, conda was chugging along and getting a lot of momentum in the
> Scientific community. And the core thing here is that conda was designed
> from the ground up to support essentially anything, This means is supports
> python packages that depend on non-python packages, but also supports
> packages that have nothing to do with python (Perl, command line tools,
> what have you...)
>
> So I have been focusing on conda lately.
>

The whole reason I started this discussion is to make sure wheel has a
standard way to do what is needed for those usecases.

conda, rpm, deb, or eggs as used in enthought are all essentially the same:
an archive with a bunch of metadata. The real issue is standardising on the
exact formats. As you noticed, there is not much missing in the wheel
*spec* to get most of what's needed. We've used eggs for that purpose for
almost 10 years at Enthought, and we did not need that many extensions on
top of the egg format after all.



> Which brings me back to the question: should the python tools (i.e. wheel)
> be extended to support more use-cases, specifically non-python
> dependencies? Or do we just figure that that's a problem better solved by
> projects with a larger scope (i.e. rpm, deb, conda, canopy).
>

IMO, given that wheels do most of what's needed, it is worth supporting
most simple usecases (compiled libraries required by well known
extensions). Right now, such packages (pyzmq, numpy, cryptography, lxml)
resort to quite horrible custom hacks to support those cases.

Hope that clarifies the intent,

David



> I'm on the fence here. I mostly care about Python, and I think we're
> pretty darn close with allowing wheel to support the non-python
> dependencies, which would allow us all to "simply pip install" pretty much
> anything -- that would be cool. But maybe it's a bit of a slippery slope,
> and if we go there, we'll end up re-writing conda.
>
> BTW, while you can't generally install a conda package in/for another
> python, you can generally install a wheel in a conda python....There are a
> few issues with pip/setuptools trying to resolve dependencies while not
> knowing about conda packages, but it does mostly work.
>
> Not sure that helped the discussion -- but I've been wrestling with this
> for a while, so thought I'd get my thoughts out there.
>
>
> -Chris
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150413/3c24610b/attachment-0001.html>