[Distutils] Making pip and PyPI work with conda packages

Donald Stufft donald at stufft.io
Sun May 17 00:03:13 CEST 2015

> On May 16, 2015, at 3:04 PM, David Mertz <dmertz at continuum.io> wrote:
> I've just started monitoring this SIG to get a sense of the issues and status of things.  I've also just started working for Continuum Analytics.
> Continuum has a great desire to make 'pip' work with conda packages.  Obviously, we love for users to choose the Anaconda Python distribution but many will not for a variety of reasons (many good reasons).
> However, we would like for users of other distros still to be able to benefit from our creation of binary packages for many platforms in the conda format.  As has been discussed in recent threads on dependency solving, the way conda provides metadata apart from entire packages makes much of that work easier.  But even aside from that, there are simply a large number of well-tested packages (not only for Python, it is true, so that's possibly a wrinkle in the task) we have generated in conda format.
> It is true that right now, a user can in principle type:
>   % pip install conda
>   % conda install some_conda_package
> But that creates two separate systems for tracking what's installed and what dependencies are resolved; and many users will not want to convert completely to conda after that step.
> What would be better as a user experience would be to let users do this:
>   % pip install --upgrade pip
>   % pip install some_conda_package
> Whether that second command ultimately downloads code from pyip.python.org <http://pyip.python.org/> or from repo.continuum.io <http://repo.continuum.io/> is probably less important for a user experience perspective.  Continuum is very happy to upload all of our conda packages to PyPI if this would improve this user experience.  Obviously, the idea here would be that the user would be able to type 'pip list' and friends afterward, and have knowledge of what was installed, even as conda packages.
> I'm hoping members of the SIG can help me understand both the technical and social obstacles that need to be overcome before this can happen.

As Paul mentioned, I’m not sure I see a major benefit to being able to ``pip install`` a conda package that doesn’t come with a lot of footguns, since any conda package either won’t be able to depend on things like Python or random C libraries or we’re going to have to just ignore those dependencies or what have you. I think a far more workable solution is one that translates a conda package to a Wheel.

Practically speaking the only real benefit that conda packages has over pip is the one benefit that simply teaching pip to install conda packages won’t provide - Namely that it supports things which aren’t Python packages. However I don’t think it’s likely that we’re going to be able to install R or erlang or whatever into a virtual environment (for instance), but maybe I’m wrong. There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting, it just hasn’t gotten them yet because either we have to convince people to publish our new formats (e.g. we can’t go out and create a wheel repo of common packages) or because we haven’t gotten to it yet because dealing with the crushing legacy of PyPI’s ~400k packages is significant slow down factor.

Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150516/95b08edc/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150516/95b08edc/attachment.sig>

More information about the Distutils-SIG mailing list