[Distutils] Handling the binary dependency management problem

[Chris Barker]

On Mon, Dec 2, 2013 at 5:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> And the conda folks are working on playing nice with virtualenv - I don't
> we'll see a similar offer from Microsoft for MSI any time soon :)
>
nice to know...

> > > a single organisation. Pip (when used normally) communicates with PyPI
> > > and no single organisation controls the content of PyPI.
>
can't you point pip to a "wheelhouse'? How is that different?

>>For built distributions they could do
> > > the same - except that pip/PyPI don't provide a mechanism for them to
> > > do so.
>
I'm still confused as to what conda provides here -- as near as I can tell,
conda has a nice hash-based way to ensure binary compatibility -- which is
a good thing. But the "curated set of packages" is an independent issue.
What's stopping anyone from creating a nice curated set of packages with
binary wheels (like the Gohlke repo....)

And wouldn't it be better to make wheel a bit more robust in this regard
than add yet another recommended tool to the mix?

> Exactly, this is the difference between pip and conda - conda is a
> solution for installing from curated *collections* of packages. It's
> somewhat related to the tagging system people are speculating about for
> PyPI, but instead of being purely hypothetical, it already exists.
>
Does it? I only know of one repository of conda packages -- and it provides
poor support for some things (like wxPython -- does it support any desktop
GUI on OS-X?)

So why do we think that conda is a better option for these unknown curatied
repos?

Also, I'm not sure I WANT anymore curated repos -- I'd rather a standard
set by python.org that individual package maintainers can choose to support.

PyPI wheels would then be about publishing "default" versions of
> components, with the broadest compatibility, while conda would be a
> solution for getting access to alternate builds that may be faster, but
> require external shared dependencies.
>
I'm still confused as to why packages need to share external dependencies
(though I can see why it's nice...) .

But what's the new policy here? Anaconda and Canopy exist already? Do we
need to endorse them? Why? If you want "PyPI wheels would then be about
publishing "default" versions of components, with the broadest
compatibility," -- then we still need to improve things a bit, but we can't
say "we're done"

What Christoph is doing is producing a cross-platform curated binary
> software stack, including external dependencies. That's precisely the
> problem I'm suggesting we *not* try to solve in the core tools any time
> soon, but instead support bootstrapping conda to solve the problem at a
> different layer.
>
So we are advocating that others, like Christoph, create curated stack with
conda? Asside from whether conda really provides much more than wheel to
support doing this, I think it's a BAD idea to encourage it: I'd much
rather encourage package maintainers to build "standard" packages, so we
can get some extra interoperabilty.

Example: you can't use wxPython with Anocoda (on the Mac, anyway). At least
not without figuring out how to build it yourself, an I'm not sure it will
even work then. (and it is a fricking nightmare to build). But it's getting
harder to find "standard" packages for the mac for the SciPy stack, so
people are really stuck.

So the pip compatible builds for those tools would likely miss out on some
> of the external acceleration features,
>
that's fine -- but we still need those pip compatible builds ....

and the nice thing about pip-compatible builds (really
python.orgcompatible builds...) is that they play well with the other
binary
installers --

> By ceding the "distribution of cross-platform curated software stacks with
> external binary dependencies" problem to conda, users would get a solution
> to that problem that they can use *now*,
>
Well, to be fair, I've been starting a project to provide binaries for
various packages for OS_X amd did intend to give conda a good look-see, but
I really has hoped that wheels where "the way" now...oh well.
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131202/fe8bd7d3/attachment-0001.html>