[Distutils] Handling the binary dependency management problem

[Ralf Gommers]

On Mon, Dec 2, 2013 at 12:38 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 1 December 2013 22:17, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> > For example, I installed Nikola into a virtualenv last night. That
> required
> > installing the development headers for libxml2 and libxslt, but the error
> > that tells you that is a C compiler one.
> >
> > I've been a C programmer longer than I have been a Python one, but I
> still
> > had to resort to Google to try to figure out what dev libraries I needed.
>
> But that's a *build* issue, surely? How does that relate to installing
> Nikola from a set of binary wheels?
>
> I understand you are thinking about non-Python libraries, but all I
> can say is that this has *never* been an issue to my knowledge in the
> Windows world. People either ship DLLs with the Python extension, or
> build statically. I understand that things are different in the Unix
> world, but to be blunt why should Windows users care?
>
> > Outside the scientific space, crypto libraries are also notoriously hard
> to
> > build, as are game engines and GUI toolkits. (I guess database bindings
> > could also be a problem in some cases)
>
> Build issues again...
>
> > We have the option to leave handling the arbitrary binary dependency
> problem
> > to platforms, and I think we should take it.
>
> Again, can we please be clear here? On Windows, there is no issue that
> I am aware of. Wheels solve the binary distribution issue fine in that
> environment (I know this is true, I've been using wheels for months
> now - sure there may be specialist areas that need some further work
> because they haven't had as much use yet, but that's details)
>
> > This is why I suspect there will be a better near term effort/reward
> > trade-off in helping the conda folks improve the usability of their
> platform
> > than there is in trying to expand the wheel format to cover arbitrary
> binary
> > dependencies.
>
> Excuse me if I'm feeling a bit negative towards this announcement.
> I've spent many months working on, and promoting, the wheel + pip
> solution, to the point where it is now part of Python 3.4. And now
> you're saying that you expect us to abandon that effort and work on
> conda instead? I never saw wheel as a pure-Python solution, installs
> from source were fine for me in that area. The only reason I worked so
> hard on wheel was to solve the Windows binary distribution issue. If
> the new message is that people should not distribute wheels for (for
> example) lxml, pyyaml, pymzq, numpy, scipy, pandas, gmpy, and pyside
> (to name a few that I use in wheel format relatively often) then
> effectively the work I've put in has been wasted.
>

Hi, scipy developer here. In the scientific python community people are
definitely interested in and intending to standardize on wheels. Your work
on wheel + pip is much appreciated.

The problems above that you say are "build issues" aren't really build
issues (where build means what distutils/bento do to build a package).
Maybe the following concepts, shamelessly stolen from the thread linked
below, help:
- *build systems* handle the actual building of software, eg Make, CMake,
distutils, Bento, autotools, etc
- *package managers* handle the distribution and installation of built (or
source) software, eg pip, apt, brew, ports
- *build managers* are separate from the above and handle the automatic(?)
preparation of packages from the results of build systems

Conda is a package manager to the best of my understanding, but because it
controls the whole stack it can also already do parts of the job of a build
manager. This is not something that pip aims to do. Conda is fairly new and
not well understood in our community either, but maybe this (long) thread
helps:
https://groups.google.com/forum/#!searchin/numfocus/build$20managers/numfocus/mVNakFqfpZg/6h_SldGNM-EJ.


Regards,
Ralf


I'm hoping I've misunderstood here. Please clarify. Preferably with
> specifics for Windows (as "conda is a known stable platform" simply
> isn't true for me...) - I accept you're not a Windows user, so a
> pointer to already-existing documentation is fine (I couldn't find any
> myself).
>
> Paul.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131202/32cf2d02/attachment-0001.html>