[Distutils] Handling the binary dependency management problem

[Daniel Holth]

On Tue, Dec 3, 2013 at 4:13 PM, Donald Stufft <donald at stufft.io> wrote:
> I think Wheels are the way forward for Python dependencies. Perhaps not for
> things like fortran. I hope that the scientific community can start
> publishing wheels at least in addition too.
>
> I don't believe that Conda will gain the mindshare that pip has outside of
> the scientific community so I hope we don't end up with two systems that
> can't interoperate.
>
> On Dec 2, 2013, at 7:00 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>
> Side note about naming:
>
> I'm no expert, but I'm pretty sure "Anoconda" is a python distribution --
> python itself and set of pre-build packages. "conda" is the package manager
> that is used by Anoconda -- kind of like rpm is used by RedHat -- conda is
> an open-source project, and thus could be used by any of us completely apart
> from the Anoconda distribution.
>
>
> On Sun, Dec 1, 2013 at 3:38 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> > had to resort to Google to try to figure out what dev libraries I
>> > needed.
>>
>> But that's a *build* issue, surely? How does that relate to installing
>> Nikola from a set of binary wheels?
>>
> Exactly -- I've mostly dealt with this for OS-X -- there are a cadre of
> users that want binaries, and want them to "just work" -- we've had mpkg
> packages for a good while, analogous to Windows installers. Binary eggs
> never worked quite right, 'cause setuptools didn't understand "universal"
> binaries -- but it wasn't that far from working. Not really tested much yet,
> but it ;looks like binary wheels should be just fine. The concern there is
> that someone will be running, say, a homebrew-built python, and accidentally
> install a binary wheel built for the python.org python -- we should address
> that with better platform tags (and making sure pip at least give a warning
> if you try to install an incompatible wheel...)

We are at least as worried about the homebrew user uploading a popular
package as a binary wheel, and having it fail to work for the more
common? non-homebrew user.

> So what problem are we trying to solve here?
>
> 1) It's still a pain to actually build the packages -- similarly to Windows,
> you really need to build the dependent libraries statically and link them in
> - and you need to make sure that you build them with teh right sdk, and
> universally -- this is hard to do right.
>   - does Conda help you do any of that???
>
> 2) non-python binary dependencies: As it turns out, a number of python
> packages depend on the same third-party non-python dependencies: I have
> quite a few that use libpng, libfreetype, libhdf, ??? currently if you want
> to distribute binary python packages, you need to statically link or supply
> the dlls, so we end up with multiple coples of the same lib -- is this a
> problem? Maybe not -- memory is pretty cheap these days, and maybe different
> packages actually rely on different versions of the dependencies -- this
> way, at least the package builder controls that.
>
> Anoconda (the distribution  seems to address this by having conda packages
> that are essentially containers for the shared libs, and other packages that
> need those libs depend on them. I like this method, but it seems to me to be
> more a feature of the Anoconda distribution than the conda package manager
> -- in fact, I've been thinking of doing this exact same thing with binary
> wheels -- I haven't tried it yet, but don't see why it wouldn't work.
>
>> I understand you are thinking about non-Python libraries, but all I
>> can say is that this has *never* been an issue to my knowledge in the
>> Windows world.
>
>
> yes, it's a HUGE issue in the Windows world -- in fact such a huge issue
> that almost non one ever tries to build things themselves, or build a
> different python distro -- so, in fact, when someone does make a binary,
> it's pretty likely to work. But those binaries are a major pain to build!
>
> (by the way, over on python-dev there has been a recent discussion about
> stackless building a new python2.7 windows binary with a newer MS compiler
> -- which will then create exacty these issues...)
>
>> > Outside the scientific space, crypto libraries are also notoriously hard
>> > to
>> > build, as are game engines and GUI toolkits. (I guess database bindings
>> > could also be a problem in some cases)
>>
>> Build issues again...
>
>
> Yes, major ones.
>
> (another side note: you can't get wxPython for OS-X to work with Anoconda --
> there is no conda binary package, and python itself is not built in a way
> that it can access the window manager ... so no, this stuff in NOT suddenly
> easier with conda.)
>
>> Again, can we please be clear here? On Windows, there is no issue that
>> I am aware of. Wheels solve the binary distribution issue fine in that
>> environment
>
>
> They will if/when we make sure that the wheel contains meta-data about what
> compiler (really run-time version) was used for the python build and wheel
> build -- but we should, indeed, do that.
>
>> > This is why I suspect there will be a better near term effort/reward
>> > trade-off in helping the conda folks improve the usability of their
>> > platform
>> > than there is in trying to expand the wheel format to cover arbitrary
>> > binary
>> > dependencies.
>
>
> and have yet anoto=her way to do it? AARRG! I'm also absolutely unclear on
> what conda offers that isn't quite easy to address with binary wheels. And
> it seems to need help too, so it will play better with virtualenv....
>
> If conda really is a better solution, then I suppose we could go deprecate
> wheel before it gets too much "traction"...;-) But let's please not another
> one to the mix to confuse people.
>
>> Excuse me if I'm feeling a bit negative towards this announcement.
>> I've spent many months working on, and promoting, the wheel + pip
>> solution, to the point where it is now part of Python 3.4.
>
>
> And I was really lookingn forward to it as a solution for OS-X too....
>
>> I'm hoping I've misunderstood here. Please clarify. Preferably with
>> specifics for Windows (as "conda is a known stable platform" simply
>> isn't true for me...) - I accept you're not a Windows user, so a
>> pointer to already-existing documentation is fine (I couldn't find any
>> myself).
>
>
> I appreciate the Windows viewpoint -- and I think we really do want one
> solution for all.
>
> The linux distros already do all this let them keep doing it....
>
> -Chris
>
> PS: a number of scipy-related packages have been promoting Anoconda and
> Canopy as a way to get their package without dealing with building (rather
> than say, providing binary wheels). That works great for the SciPy Stack,
> but has NOT worked well for others. My example:
>
> I'm teaching an Intro to Python class -- I really like iPython, so have been
> using it for demos and recommend it to Students. The IPython web site makes
> it look like you really need to go get Anaconda or Canopy if you want
> iPython -- so a number of my students went and did that. All well. Until I
> did the extra class on wxPython, and now I've got a bunch of students that
> have no way to install it. And they mostly had no idea that they were
> running a different Python at all...
>
> Anyway -- this is a mess, particularly on OS-X (now we have python binaries
> from: Apple, python.org, fink, macports, homebrew, Anaconda, Canopy, ???) So
> yes, we need a solution, but I think binary wheels are a pretty good one,
> and I'm not sure what conda buys us...

The most striking difference may be that conda also installs and
manages Python itself. For example, conda create -n py33 python=3.3
will download and install Python 3.3 into a new environment named
py33. This is completely different than pip which tends to run inside
the same Python environment that it's installing into.

Wheels were only designed to hold Python libraries to be installed
into site-packages, with package names registered on pypi, and wheels
were designed to allow the parts of the distribution to be relocated
at install time as in putting scripts into Scripts\ or bin/ depending
on the platform.

If you wanted to interoperate you would probably install pip with
"conda install pip" and then play with some wheels, or you could
probably create a virtualenv from your conda environment and proceed.
I would not recommend trying to package the CPython interpreter as a
wheel. It wasn't designed for that. If you try let us know what
happened :-)

It's interesting that there is so much talk about compatibility with
numpy-built-with-different-options. When I was thinking about the
problem of binary compatibility I was more concerned about dynamically
linking with different versions of system libraries (glibc) than the
ones that the pre-built [wheel] package expects.

In summary conda is very different than pip+virtualenv.