[Distutils] Making pip and PyPI work with conda packages

[Donald Stufft]

> On May 16, 2015, at 8:50 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> 
> On Sat, May 16, 2015 at 4:16 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>> On Sat, May 16, 2015 at 3:03 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>> There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting,
>> 
>> Huh? I"'m confused -- didn't we just have a big thread about how pip+wheel probably ISN'T going to handle shared libs -- that those are exactly what conda packages do provide -- aside from R and Erlange, anyway :-)
>> 
>> but it's not the packages in this case that we need -- it's the environment -- and I can't see how pip is going to provide a conda environment….
> 
> I never said pip was going to provide an environment, I said the main benefit conda has over pip, which pip will most likely not get in any reasonable time frame, is that it handles things which are not Python packages.
> 
> well, I got a bit distraced by Erlang and R -- i.e. things that have nothing to do with python packages.
> 
> libxml, on the other hand, is a lib that one might want to use with a python package -- so a bit more apropos here.
> 
> But my confusion was about: "things that conda has that pip is planning on getting" -- what are those things? Any of the stuff that conda has that really useful like handling shared libs, pip is NOT getting -- yes?


The ability to resolve dependencies with static metadata is the major one that comes to my mind that’s specific to pip. The ability to have better build systems besides distutils/setuptools is a more ecosystem level one but that’s something we’ll get too.

As far as shared libs… beyond what’s already possible (sticking a shared lib inside of a python project and having libraries load that .dll explicitly) it’s not currently on the road map and may never be. I hesitate to say never because it’s obviously a problem that needs solved and if the Python ecosystem solves it (specific to shared libraries, not whole runtimes or other languages or what have you) then that would be a useful thing. I think we have lower hanging fruit that we need to deal with before something like that is even possibly to be on the radar though (if we ever put it on the radar).

> 
> A shared library is not a Python package so I’m not sure what this message is even saying? ``pip install lxml-from-conda`` is just going to flat out break because pip won’t install the libxml2 shared library.
> 
> exactly -- if you're going to install a shared lib, you need somewhere to put it -- and that's what a conda environment provides.
> 
> Trying not to go around in circles, but python _could_ provide a standard place in which to put shared libs -- and then pip _could_ provide a way to manage them. That would require dealing with that whole binary API problem, so we probably won't do it. I'm not sure what the point of contention is here:
> 
> I think it would be useful to have a way to manage shared libs solely for python packages to use -- and it would be useful to that way to be part of the standard python ecosytem. Others may not think it would be useful enough to be worth the pain in the neck it would be.
> 
> And that's what the nifty conda packages continuum (and others) have built could provide -- those shared libs that are built in a  compatible way with a python binary. After all, pure python packages are no problem, compiled python packages without any dependencies are little problem. The hard part is those darn third party libs.
> 
> conda also provides a way to mange all sorts of other stuff that has nothing to do with python, but I'm guessing  that's not what continuum would like to contribute to pypi….

I guess I’m confused what the benefit of making pip able to install a conda package would be. If Python adds someplace for shared libs to go then we could just add shared lib support to Wheels, it’s just another file type so that’s not a big deal. The hardest part is dealing with ABI compatibility. However, given the current state of things, what’s the benefit of being able to do ``pip install conda-lxml``? Either it’s going to flat out break or you’re going to have to do ``conda install libxml2`` first, and if you’re doing ``conda install libxml2`` first then why not just do ``conda install lxml``?

I view conda the same way I view apt-get, yum, Chocolatey, etc. It provides an environment and you can install a Python package into that environment, but that pip shouldn’t know how to install a .deb or a .rpm or a conda package because those packages rely on specifics to that environment and Python packages can’t.


---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150516/86247882/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150516/86247882/attachment-0001.sig>