[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Paul Moore]

On 19 May 2015 at 17:28, Chris Barker <chris.barker at noaa.gov> wrote:
> On Tue, May 19, 2015 at 5:21 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> If conda did everything pip did (and that includes consuming wheels
>> from PyPI, not just sdists, and it includes caching of downloads,
>> autobuilding of wheels etc, etc.)
>
> hmm...what about half-way -- conda does everything pip does, but not
> necessarily the same way -- i.e. you do a "conda install this_package", and
> it works for every package ( OK -- almost every ;-) ) that pip install works
> for.

Sure. Doesn't have to be the same way, but the user experience has to
be the same.

> But maybe that's not going to cut it -- in a way, we are headed there now,
> with a contingent of people porting pypi packages to conda. So far it's
> various subsets of the scientific community, but if we could get a few web
> developers to join in...

Unless project owners switch to providing conda packages, isn't there
always going to be a lag? If a new version of lxml comes out, how long
must I wait for "the conda folks" to release a package for it?

>> then I'd certainly consider how to
>> switch to conda (*not* Anaconda - I'd use a different package manager,
>> but not a different Python distribution) rather than pip.
>
> hmm -- that's the interesting technical question -- conda works at a higher
> level than pip -- it CAN manage python itself -- I'm not sure it is HAS to,
> but that's how it is usually used, and the idea is to provide a complete
> environment, which does include python itself.

Yes. But I don't want to use Anaconda Python, Same reason - how long
do I wait for the new release of Python to be available in Anaconda?
There's currently no Python 3.5 alpha for example...

>> But "considering switching" would include getting PyPI supporting
>> conda packages,
>
> uhm, why? if there is a community supported repo of packages -- why does it
> have to be PyPi?

If conda/binstar is good enough to replace pip/PyPI, there's no reason
for pip/PyPI to still exist. So in effect binstar *becomes* PyPI.

There's an element of evangelisation going on here - you're
(effectively) asking what it'd take to persuade me to use conda in
place of pip. I'm playing hard to get, a little, because I see no
specific benefits to me in using conda, so I don't see why I should
accept any loss at all, in the absence of a benefit to justify it.

My biggest worry is that at some point, "if you want numpy/scipy, you
should use conda" becomes an explicit benefit of conda, and pip/PyPI
users get abandoned by the scientific community. If that happens, I'd
rather see the community rally behind conda than see a split. But I
hope that's not the way things end up going.

>> getting ensurepip replaced with ensureconda, etc. A
>> total replacement for pip, in other words.

This is the key point. The decision was made to "bless" pip as the
official Python package manager. Should we revisit that decision? If
not, then how do we ensure that pip (and the surrounding
infrastructure) handles the needs of the *whole* Python community? If
the authors of scientific extensions for Python abandon pip for conda,
then pip isn't supporting that part of the community properly. But
conversely, if the scientific community doesn't look to address their
issues within the pip/wheel infrastructure, how can we do anything to
avoid a rift?

(end of doom and gloom section ;-))

>> As a pip maintainer I'm obviously biased, but if conda is intending to
>> replace pip as the official packaging solution for Python, then it
>> needs to do so completely. If it doesn't do that, then we (PyPA and
>> the Python core developers) need to be able to credibly say that pip
>> is the official solution, and that means that we need to make sure
>> that pip/wheel provides the best user experience possible. That
>> includes persuading parts of the Python community (e.g. Scientific
>> users) not to abandon the standard solution in favour of a custom one.
>
>
> I agree here. Though we do have a problem -- as Nick has pointed out, the
> "full" scientific development process -- even if python-centered, requires
> non-python parts, even beyond shared libs -- a fortran compiler is a big
> one, but also maybe other languages, like R, or Julia, etc. Or LLVM (for
> numba), or...
>
> This is why Continuum build conda -- they wanted to provide a way to manage
> all that.
>
> So is it possible for PyPA to grow the features to mange all the python
> bits, and then have things like conda use pip inside of Anaconda, maybe?


I'd like to think so. The goal of pip is to be the baseline Python
package manager. We're expecting Linux distributions to build their
system packages via wheel, why can't Anaconda? Part of the problem
here, to my mind, is that it's *very* hard for the outsider to
separate out (Ana)conda-as-a-platform versus conda-as-a-tool, versus
conda-as-a-distribution-format.


> or
> SOME transition where you can add conda if and only if you need its unique
> features, and as a add-it-later-to-what-you-have solution, rather than, when
> you need R or some such, you need to
>
>   * Toss out your current setup
>   * Install Anaconda (or miniconda)
>   * Switch from virtualenv to conda environments
>   * re-install all your dependencies

Yeah, that's hopeless. And worse still is the possibility that a pure
Python user might have to do that just to gain access to a particular
package.

I keep saying this, and I ought to ask - is there *any* likelihood
that a package would formally abandon any attempt to provide binary
distributions for Windows (or OSX, or whatever) except in conda
format? So PyPI users will be told "install from source yourself or
switch to conda". If there's no intention for that to ever happen, a
lot of the "conda vs pip" discussion is less relevant. At the moment,
there are projects like numpy that don't distribute Windows wheels on
PyPI, but Christoph Gohlke has most of them available, and in general
such projects seem to be aiming to move to wheels. So there aren't any
practical cases of "conda-only" packages.

> And for even that to work, we need a way for everything installable by pip
> to be installable within that conda environment -- which we could probably
> achieve.

See above - without some serious resource on the conda side, that
"everything" is unlikely.

>> My fear here is a split in the Python community, with some packages
>> only being available via one ecosystem, and some via another.
>
>
> Exactly. While I'm not at all sure that we could get to a "one way to do it"
> that would meet every community's needs,  I do think that we could push
> pip+pypi+wheel a little further to better support at least the
> python-centric stuff -- i.e. third party libs, which would get us a lot
> farther.

Agreed. Understanding the *actual* problem here is important though
(see my other post about clarifying why dynamic linking is so
important).

> And again, it's not just the scipy stack -- there is stuff like image
> manipulation packages, etc, that could be better handled.

Well, (for example) Pillow provides wheels with no issue, so I'm not
sure what you're thinking of here?

> And the geospatial packages are a mess, too - is that "scientific"? -- I
> don't know, but it's the "new hotness" in web development.

I can't really comment on that one, as I've never used them. Does that
make me untrendy? :-)

Paul