[Distutils] Making pip and PyPI work with conda packages

[Donald Stufft]

> On May 17, 2015, at 8:12 PM, Robert Collins <robertc at robertcollins.net> wrote:
> 
> 
> On 17 May 2015 5:05 pm, "Nick Coghlan" <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> wrote:
> >
> >
> > On 18 May 2015 07:32, "Chris Barker" <chris.barker at noaa.gov <mailto:chris.barker at noaa.gov>> wrote:
> > >
> > > On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> wrote:
> > >>
> > >> >   % pip install --upgrade pip
> > >> >   % pip install some_conda_package
> > >>
> > >> This gets the respective role of the two tools reversed - it's like my
> > >> asking for "pip install some_fedora_rpm" to be made to work.
> > >
> > >
> > > I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice.
> > >
> > > The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that.
> > >
> > > And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together.
> > >
> > > Continuum has a bunch of smart people, though.
> > >
> > >> However, having conda use "pip install" in its build scripts so that
> > >> it reliably generates pip compatible installation metadata would be a
> > >> possibility worth discussing - that's what we've started doing in
> > >> Fedora, so that runtime utilities like pkg_resources can work
> > >> correctly.
> > >
> > >
> > > Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script --  so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the "offical" conda packages is the standard).
> > >
> > > But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ???
> > >
> > > I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already.
> > >
> > > So what would using pip inside a conda build script buy you that using setuptools does not?
> >
> > Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ <https://www.python.org/dev/peps/pep-0376/> compliant metadata by default.
> >
> > However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip & attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently.
> 
> That glue is actually very shallow...I think we should rip it out of pip and perhaps put it in setuptools. It's about building, not installing.
> 


So a benefit of using pip instead of setuptools is that as we move to a pluggable build system pip can act as a unified fronted to multiple build systems, instead of every system having to implement each pluggable build system themselves.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150517/dd7e3358/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150517/dd7e3358/attachment-0001.sig>