[Distutils] Handling the binary dependency management problem

[Chris Barker]

Side note about naming:

I'm no expert, but I'm pretty sure "Anoconda" is a python distribution --
python itself and set of pre-build packages. "conda" is the package manager
that is used by Anoconda -- kind of like rpm is used by RedHat -- conda is
an open-source project, and thus could be used by any of us completely
apart from the Anoconda distribution.


On Sun, Dec 1, 2013 at 3:38 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> > had to resort to Google to try to figure out what dev libraries I needed.
>
> But that's a *build* issue, surely? How does that relate to installing
> Nikola from a set of binary wheels?
>
> Exactly -- I've mostly dealt with this for OS-X -- there are a cadre of
users that want binaries, and want them to "just work" -- we've had mpkg
packages for a good while, analogous to Windows installers. Binary eggs
never worked quite right, 'cause setuptools didn't understand "universal"
binaries -- but it wasn't that far from working. Not really tested much
yet, but it ;looks like binary wheels should be just fine. The concern
there is that someone will be running, say, a homebrew-built python, and
accidentally install a binary wheel built for the python.org python -- we
should address that with better platform tags (and making sure pip at least
give a warning if you try to install an incompatible wheel...)

So what problem are we trying to solve here?

1) It's still a pain to actually build the packages -- similarly to
Windows, you really need to build the dependent libraries statically and
link them in - and you need to make sure that you build them with teh right
sdk, and universally -- this is hard to do right.
  - does Conda help you do any of that???

2) non-python binary dependencies: As it turns out, a number of python
packages depend on the same third-party non-python dependencies: I
have quite a few that use libpng, libfreetype, libhdf, ??? currently if you
want to distribute binary python packages, you need to statically link or
supply the dlls, so we end up with multiple coples of the same lib -- is
this a problem? Maybe not -- memory is pretty cheap these days, and maybe
different packages actually rely on different versions of the dependencies
-- this way, at least the package builder controls that.

Anoconda (the distribution  seems to address this by having conda packages
that are essentially containers for the shared libs, and other packages
that need those libs depend on them. I like this method, but it seems to me
to be more a feature of the Anoconda distribution than the conda package
manager -- in fact, I've been thinking of doing this exact same thing with
binary wheels -- I haven't tried it yet, but don't see why it wouldn't work.

I understand you are thinking about non-Python libraries, but all I
> can say is that this has *never* been an issue to my knowledge in the
> Windows world.


yes, it's a HUGE issue in the Windows world -- in fact such a huge issue
that almost non one ever tries to build things themselves, or build a
different python distro -- so, in fact, when someone does make a binary,
it's pretty likely to work. But those binaries are a major pain to build!

(by the way, over on python-dev there has been a recent discussion about
stackless building a new python2.7 windows binary with a newer MS compiler
-- which will then create exacty these issues...)

> Outside the scientific space, crypto libraries are also notoriously hard
> to
> > build, as are game engines and GUI toolkits. (I guess database bindings
> > could also be a problem in some cases)
>
> Build issues again...


Yes, major ones.

(another side note: you can't get wxPython for OS-X to work with Anoconda
-- there is no conda binary package, and python itself is not built in a
way that it can access the window manager ... so no, this stuff in NOT
suddenly easier with conda.)

Again, can we please be clear here? On Windows, there is no issue that
> I am aware of. Wheels solve the binary distribution issue fine in that
> environment


They will if/when we make sure that the wheel contains meta-data about what
compiler (really run-time version) was used for the python build and wheel
build -- but we should, indeed, do that.

> This is why I suspect there will be a better near term effort/reward
> > trade-off in helping the conda folks improve the usability of their
> platform
> > than there is in trying to expand the wheel format to cover arbitrary
> binary
> > dependencies.
>

and have yet anoto=her way to do it? AARRG! I'm also absolutely unclear on
what conda offers that isn't quite easy to address with binary wheels. And
it seems to need help too, so it will play better with virtualenv....

If conda really is a better solution, then I suppose we could
go deprecate wheel before it gets too much "traction"...;-) But let's
please not another one to the mix to confuse people.

Excuse me if I'm feeling a bit negative towards this announcement.
> I've spent many months working on, and promoting, the wheel + pip
> solution, to the point where it is now part of Python 3.4.


And I was really lookingn forward to it as a solution for OS-X too....

I'm hoping I've misunderstood here. Please clarify. Preferably with
> specifics for Windows (as "conda is a known stable platform" simply
> isn't true for me...) - I accept you're not a Windows user, so a
> pointer to already-existing documentation is fine (I couldn't find any
> myself).
>

I appreciate the Windows viewpoint -- and I think we really do want one
solution for all.

The linux distros already do all this let them keep doing it....

-Chris

PS: a number of scipy-related packages have been promoting Anoconda and
Canopy as a way to get their package without dealing with building (rather
than say, providing binary wheels). That works great for the SciPy Stack,
but has NOT worked well for others. My example:

I'm teaching an Intro to Python class -- I really like iPython, so have
been using it for demos and recommend it to Students. The IPython web site
makes it look like you really need to go get Anaconda or Canopy if you want
iPython -- so a number of my students went and did that. All well. Until I
did the extra class on wxPython, and now I've got a bunch of students that
have no way to install it. And they mostly had no idea that they were
running a different Python at all...

Anyway -- this is a mess, particularly on OS-X (now we have python binaries
from: Apple, python.org, fink, macports, homebrew, Anaconda, Canopy, ???)
So yes, we need a solution, but I think binary wheels are a pretty good
one, and I'm not sure what conda buys us...



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131202/24254671/attachment-0001.html>