[Distutils] Handling the binary dependency management problem

Ralf Gommers ralf.gommers at gmail.com
Mon Dec 2 08:20:08 CET 2013

On Mon, Dec 2, 2013 at 12:38 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 1 December 2013 22:17, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > For example, I installed Nikola into a virtualenv last night. That
> required
> > installing the development headers for libxml2 and libxslt, but the error
> > that tells you that is a C compiler one.
> >
> > I've been a C programmer longer than I have been a Python one, but I
> still
> > had to resort to Google to try to figure out what dev libraries I needed.
> But that's a *build* issue, surely? How does that relate to installing
> Nikola from a set of binary wheels?
> I understand you are thinking about non-Python libraries, but all I
> can say is that this has *never* been an issue to my knowledge in the
> Windows world. People either ship DLLs with the Python extension, or
> build statically. I understand that things are different in the Unix
> world, but to be blunt why should Windows users care?
> > Outside the scientific space, crypto libraries are also notoriously hard
> to
> > build, as are game engines and GUI toolkits. (I guess database bindings
> > could also be a problem in some cases)
> Build issues again...
> > We have the option to leave handling the arbitrary binary dependency
> problem
> > to platforms, and I think we should take it.
> Again, can we please be clear here? On Windows, there is no issue that
> I am aware of. Wheels solve the binary distribution issue fine in that
> environment (I know this is true, I've been using wheels for months
> now - sure there may be specialist areas that need some further work
> because they haven't had as much use yet, but that's details)
> > This is why I suspect there will be a better near term effort/reward
> > trade-off in helping the conda folks improve the usability of their
> platform
> > than there is in trying to expand the wheel format to cover arbitrary
> binary
> > dependencies.
> Excuse me if I'm feeling a bit negative towards this announcement.
> I've spent many months working on, and promoting, the wheel + pip
> solution, to the point where it is now part of Python 3.4. And now
> you're saying that you expect us to abandon that effort and work on
> conda instead? I never saw wheel as a pure-Python solution, installs
> from source were fine for me in that area. The only reason I worked so
> hard on wheel was to solve the Windows binary distribution issue. If
> the new message is that people should not distribute wheels for (for
> example) lxml, pyyaml, pymzq, numpy, scipy, pandas, gmpy, and pyside
> (to name a few that I use in wheel format relatively often) then
> effectively the work I've put in has been wasted.

Hi, scipy developer here. In the scientific python community people are
definitely interested in and intending to standardize on wheels. Your work
on wheel + pip is much appreciated.

The problems above that you say are "build issues" aren't really build
issues (where build means what distutils/bento do to build a package).
Maybe the following concepts, shamelessly stolen from the thread linked
below, help:
- *build systems* handle the actual building of software, eg Make, CMake,
distutils, Bento, autotools, etc
- *package managers* handle the distribution and installation of built (or
source) software, eg pip, apt, brew, ports
- *build managers* are separate from the above and handle the automatic(?)
preparation of packages from the results of build systems

Conda is a package manager to the best of my understanding, but because it
controls the whole stack it can also already do parts of the job of a build
manager. This is not something that pip aims to do. Conda is fairly new and
not well understood in our community either, but maybe this (long) thread


I'm hoping I've misunderstood here. Please clarify. Preferably with
> specifics for Windows (as "conda is a known stable platform" simply
> isn't true for me...) - I accept you're not a Windows user, so a
> pointer to already-existing documentation is fine (I couldn't find any
> myself).
> Paul.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131202/32cf2d02/attachment-0001.html>

More information about the Distutils-SIG mailing list