[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

Chris Barker chris.barker at noaa.gov
Tue May 19 18:28:18 CEST 2015

On Tue, May 19, 2015 at 5:21 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> If conda did everything pip did (and that includes consuming wheels
> from PyPI, not just sdists, and it includes caching of downloads,
> autobuilding of wheels etc, etc.)

hmm...what about half-way -- conda does everything pip does, but not
necessarily the same way -- i.e. you do a "conda install this_package", and
it works for every package ( OK -- almost every ;-) ) that pip install
works for.

But maybe that's not going to cut it -- in a way, we are headed there now,
with a contingent of people porting pypi packages to conda. So far it's
various subsets of the scientific community, but if we could get a few web
developers to join in...

then I'd certainly consider how to
> switch to conda (*not* Anaconda - I'd use a different package manager,
> but not a different Python distribution) rather than pip.

hmm -- that's the interesting technical question -- conda works at a higher
level than pip -- it CAN manage python itself -- I'm not sure it is HAS to,
but that's how it is usually used, and the idea is to provide a complete
environment, which does include python itself.

> But "considering switching" would include getting PyPI supporting
> conda packages,

uhm, why? if there is a community supported repo of packages -- why does it
have to be PyPi?

> getting ensurepip replaced with ensureconda, etc. A
> total replacement for pip, in other words.

> As a pip maintainer I'm obviously biased, but if conda is intending to
> replace pip as the official packaging solution for Python, then it
> needs to do so completely. If it doesn't do that, then we (PyPA and
> the Python core developers) need to be able to credibly say that pip
> is the official solution, and that means that we need to make sure
> that pip/wheel provides the best user experience possible. That
> includes persuading parts of the Python community (e.g. Scientific
> users) not to abandon the standard solution in favour of a custom one.

I agree here. Though we do have a problem -- as Nick has pointed out, the
"full" scientific development process -- even if python-centered, requires
non-python parts, even beyond shared libs -- a fortran compiler is a big
one, but also maybe other languages, like R, or Julia, etc. Or LLVM (for
numba), or...

This is why Continuum build conda -- they wanted to provide a way to manage
all that.

So is it possible for PyPA to grow the features to mange all the python
bits, and then have things like conda use pip inside of Anaconda, maybe? or
SOME transition where you can add conda if and only if you need its unique
features, and as a add-it-later-to-what-you-have solution, rather than,
when you need R or some such, you need to

  * Toss out your current setup
  * Install Anaconda (or miniconda)
  * Switch from virtualenv to conda environments
  * re-install all your dependencies

And for even that to work, we need a way for everything installable by pip
to be installable within that conda environment -- which we could probably

My fear here is a split in the Python community, with some packages
> only being available via one ecosystem, and some via another.

Exactly. While I'm not at all sure that we could get to a "one way to do
it" that would meet every community's needs,  I do think that we could push
pip+pypi+wheel a little further to better support at least the
python-centric stuff -- i.e. third party libs, which would get us a lot

And again, it's not just the scipy stack -- there is stuff like image
manipulation packages, etc, that could be better handled.

And the geospatial packages are a mess, too - is that "scientific"? -- I
don't know, but it's the "new hotness" in web development.



Christopher Barker, Ph.D.

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150519/0bca4b86/attachment-0001.html>

More information about the Distutils-SIG mailing list