[Distutils] Handling the binary dependency management problem

Tue Dec 3 22:13:37 CET 2013

I think Wheels are the way forward for Python dependencies. Perhaps not for things like fortran. I hope that the scientific community can start publishing wheels at least in addition too. 

I don't believe that Conda will gain the mindshare that pip has outside of the scientific community so I hope we don't end up with two systems that can't interoperate. 

> On Dec 2, 2013, at 7:00 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> 
> Side note about naming:
> 
> I'm no expert, but I'm pretty sure "Anoconda" is a python distribution -- python itself and set of pre-build packages. "conda" is the package manager that is used by Anoconda -- kind of like rpm is used by RedHat -- conda is an open-source project, and thus could be used by any of us completely apart from the Anoconda distribution.
> 
> 
> On Sun, Dec 1, 2013 at 3:38 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>> > had to resort to Google to try to figure out what dev libraries I needed.
>> 
>> But that's a *build* issue, surely? How does that relate to installing
>> Nikola from a set of binary wheels?
> Exactly -- I've mostly dealt with this for OS-X -- there are a cadre of users that want binaries, and want them to "just work" -- we've had mpkg packages for a good while, analogous to Windows installers. Binary eggs never worked quite right, 'cause setuptools didn't understand "universal" binaries -- but it wasn't that far from working. Not really tested much yet, but it ;looks like binary wheels should be just fine. The concern there is that someone will be running, say, a homebrew-built python, and accidentally install a binary wheel built for the python.org python -- we should address that with better platform tags (and making sure pip at least give a warning if you try to install an incompatible wheel...)
> 
> So what problem are we trying to solve here?
> 
> 1) It's still a pain to actually build the packages -- similarly to Windows, you really need to build the dependent libraries statically and link them in - and you need to make sure that you build them with teh right sdk, and universally -- this is hard to do right.
>   - does Conda help you do any of that???
> 
> 2) non-python binary dependencies: As it turns out, a number of python packages depend on the same third-party non-python dependencies: I have quite a few that use libpng, libfreetype, libhdf, ??? currently if you want to distribute binary python packages, you need to statically link or supply the dlls, so we end up with multiple coples of the same lib -- is this a problem? Maybe not -- memory is pretty cheap these days, and maybe different packages actually rely on different versions of the dependencies -- this way, at least the package builder controls that.
> 
> Anoconda (the distribution  seems to address this by having conda packages that are essentially containers for the shared libs, and other packages that need those libs depend on them. I like this method, but it seems to me to be more a feature of the Anoconda distribution than the conda package manager -- in fact, I've been thinking of doing this exact same thing with binary wheels -- I haven't tried it yet, but don't see why it wouldn't work.
> 
>> I understand you are thinking about non-Python libraries, but all I
>> can say is that this has *never* been an issue to my knowledge in the
>> Windows world.
> 
> yes, it's a HUGE issue in the Windows world -- in fact such a huge issue that almost non one ever tries to build things themselves, or build a different python distro -- so, in fact, when someone does make a binary, it's pretty likely to work. But those binaries are a major pain to build!
> 
> (by the way, over on python-dev there has been a recent discussion about stackless building a new python2.7 windows binary with a newer MS compiler -- which will then create exacty these issues...)
> 
>> > Outside the scientific space, crypto libraries are also notoriously hard to
>> > build, as are game engines and GUI toolkits. (I guess database bindings
>> > could also be a problem in some cases)
>> 
>> Build issues again...
> 
> Yes, major ones.
> 
> (another side note: you can't get wxPython for OS-X to work with Anoconda -- there is no conda binary package, and python itself is not built in a way that it can access the window manager ... so no, this stuff in NOT suddenly easier with conda.)
> 
>> Again, can we please be clear here? On Windows, there is no issue that
>> I am aware of. Wheels solve the binary distribution issue fine in that
>> environment
> 
> They will if/when we make sure that the wheel contains meta-data about what compiler (really run-time version) was used for the python build and wheel build -- but we should, indeed, do that.
> 
>> > This is why I suspect there will be a better near term effort/reward
>> > trade-off in helping the conda folks improve the usability of their platform
>> > than there is in trying to expand the wheel format to cover arbitrary binary
>> > dependencies.
> 
> and have yet anoto=her way to do it? AARRG! I'm also absolutely unclear on what conda offers that isn't quite easy to address with binary wheels. And it seems to need help too, so it will play better with virtualenv....
> 
> If conda really is a better solution, then I suppose we could go deprecate wheel before it gets too much "traction"...;-) But let's please not another one to the mix to confuse people.
> 
>> Excuse me if I'm feeling a bit negative towards this announcement.
>> I've spent many months working on, and promoting, the wheel + pip
>> solution, to the point where it is now part of Python 3.4.
> 
> And I was really lookingn forward to it as a solution for OS-X too....
> 
>> I'm hoping I've misunderstood here. Please clarify. Preferably with
>> specifics for Windows (as "conda is a known stable platform" simply
>> isn't true for me...) - I accept you're not a Windows user, so a
>> pointer to already-existing documentation is fine (I couldn't find any
>> myself).
> 
> I appreciate the Windows viewpoint -- and I think we really do want one solution for all.
> 
> The linux distros already do all this let them keep doing it....
> 
> -Chris
> 
> PS: a number of scipy-related packages have been promoting Anoconda and Canopy as a way to get their package without dealing with building (rather than say, providing binary wheels). That works great for the SciPy Stack, but has NOT worked well for others. My example:
> 
> I'm teaching an Intro to Python class -- I really like iPython, so have been using it for demos and recommend it to Students. The IPython web site makes it look like you really need to go get Anaconda or Canopy if you want iPython -- so a number of my students went and did that. All well. Until I did the extra class on wxPython, and now I've got a bunch of students that have no way to install it. And they mostly had no idea that they were running a different Python at all...
> 
> Anyway -- this is a mess, particularly on OS-X (now we have python binaries from: Apple, python.org, fink, macports, homebrew, Anaconda, Canopy, ???) So yes, we need a solution, but I think binary wheels are a pretty good one, and I'm not sure what conda buys us...
> 
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chris.Barker at noaa.gov
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131203/0fc2d460/attachment.html>