[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Paul Moore]

Sigh. I really am going to try to stop monopolising this thread - but
you keep making good points I feel I "have" to respond to :-) I'll try
to keep to essentials.

On 19 May 2015 at 22:11, Chris Barker <chris.barker at noaa.gov> wrote:
> On Tue, May 19, 2015 at 10:58 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> you can grab and build the latest Python3.5 inside a conda environment just
> as well. Or are you using python.org builds for alpha versions, too?

Yep, I basically never build Python myself, except when developing it.

> Oh, and as a conda environment sits at a higher level than python, it's
> actually easier to set up an environment specifically for a particular
> version of python.
>
> And anyone could put up a conda package of Python3.5 Alpha as well --- once
> the build script is written, it's pretty easy. But again -- teh more than
> one way to do it problem.

Yeah, I'm trying to never build anything for myself, just consume
binaries. Having all binaries built by "the conda people" is a
bottleneck. Having pip auto-build wheels once and reuse them (coming
in the next version, yay!) is good enough. Having projects upload
wheels to PyPI is ideal. Building wheels myself from wininsts provided
by others or by doing the awkward work once and hosting them in a
personal index is acceptable.

I've spent too much of my life trying to build other people's C code.
I'd like to stop now :-)

My needs don't extend to highly specialised *and* hard to build stuff.
The specialised stuff I care about is usually not hard to build, and
the hard to build stuff is usually relatively mainstream (basic numpy
and scipy as opposed to the specialised scientific stuff)

> We may want to look back at a thread on this list where Travis Oliphant
> talks about why he built conda, etc. (I can't find it now -- maybe someone
> with better google-fu than me can. It think it was a thread on this list,
> probably about a year ago)
>
> or read his Blog Post:
>
> http://technicaldiscovery.blogspot.com/2013/12/why-i-promote-conda.html

Thanks for the link - I'll definitely read that.

> One of the key points is that when they started building conda -- pip+wheel
> where not mature, and the core folks behind them didn't want to support what
> was needed (dynamic libs, etc) -- and still don't.

Well, I'm not sure "don't want to" is accurate these days. "Think the
problem is harder than you're making it sound" may be accurate, as
might "have no resource to spare to do the work, but would like others
to".

>> My biggest worry is that at some point, "if you want numpy/scipy, you
>> should use conda" becomes an explicit benefit of conda,
>
> That is EXACTLY what the explicit benefit of conda is. I think we'll get
> binary wheels for numpy and scipy up on PyPi before too long, but the rest
> of the more complex stuff is not going to be there.

I did specifically mean numpy and scipy. People are using those, and
pandas and matplotlib, for a lot of non-scientific things (business
data analysis as a replacement for Excel, for example). Forcing such
people to use Anaconda seems like a mistake - the scientific community
and the business community have very different perspectives.

Conceded, numpy/scipy are *currently* hard to get - we're in the
middle of a transition from wininst (which numpy/scipy did supply) to
wheel. But the *intention* is that wheels will be an acceptable
replacement for wininst installers, so giving up before we reach that
goal would be a shame.

>> This is the key point. The decision was made to "bless" pip as the
>> official Python package manager. Should we revisit that decision?
>
> I'm not sure I want to be the one to bring that up ;-)

Well, Nick Coghlan is probably better placed to comment on that. I
don't really understand his vision for pip and wheel as something that
other tools build on - and whether or not the fact that the scientific
community feels the need to build a completely independent
infrastructure is a problem in that context. But if it is a problem,
maybe that decision *should* be reviewed.

> When you say "build their system packages via wheel" -- what does that mean?
> and why wheel, rather than, say, pip + setuptools?

Again, that's something for Nick to comment on - I don't know how
wheel (it's wheel more than pip in this context, I believe) fits into
RPM/deb building processes. But I do know that's how he's been talking
about things working. I can't see why conda would be any different.

> you can put whatever you want in a conda build script -- the current
> standard practice baseline for python packages is:
>
> $PYTHON setup.py install

That sounds like conda doesn't separate build from install - is there
a conda equivalent of a "binary distribution" like a wheel? There are
a lot of reasons why pip/wheel is working hard to move to a separation
of build and install steps, and if conda isn't following that, I'd
like to know its response to those issues. (I don't know what all the
issues are - it'd take some searching of the list archives to find
previous discussions. One obvious one is "we don't want end users to
have to have a compiler").

> it's that simple. And I don't know if this is what it actually does, but
> essentially the conda package is all the stuff that that script added to the
> environment. So if changing that invocation to use pip would get us some
> better meta data or what have you, then by all means, we should change that
> standard of practice.

Well, it sounds like using "setup.py bdist_wheel" and then repackaging
up the contents of the wheel might be easier than playing "spot what
changed" with a setup.py install invocation. But I'm no expert. (If
the response to that is "wheel doesn't handle X" then I'd rather see
someone offering to help *improve* wheel to handle that situation!)

> (but it seems like an odd thing to have to use the package manager to build
> the package correctly -- shouldn't' that be distutils or setuptools' job?

Wheels can be built using setuptools (bdist_wheel) or just packaging
up some files manually. All pip does in this context is orchestrate
the running of setup.py bdist_wheel. You lot use "conda" to mean the
format, the package manager, and the distribution channel - give me
some slack if I occasionally use "pip" when I mean "wheel" or
"setuptools" :-) :-)

> Sadly, we are already there for minor packages, at least. Oh wait, not so
> minor -- the geospatial stack is not well supported on PyPi. I don't think
> there are pynetcdf or pyhdf binaries, etc...

There is a point where "specialised enough to only matter to
Acaconda's target audience" is OK. And Christoph Gohlke's
distributions fill a huge gap (losing that because "you should use
conda" would be a huge issue, IMO).

>> lot of the "conda vs pip" discussion is less relevant. At the moment,
>> there are projects like numpy that don't distribute Windows wheels on
>> PyPI, but Christoph Gohlke has most of them available,
>
> yes, but in a form that is not redistributable on PyPi...

Well, it's not clear to me how insurmountable that problem is - his
page doesn't specifically mention redistribution limitations (I
presume it's license issues?) Has anyone discussed the possibility of
addressing the issues? If it is licensing (of MKL?) then could the
ones without license implications be redistributed? Could the PSF
assist with getting a redistribution license for the remainder? I've
no idea what the issues are - I'm just asking if there are approaches
no-one has considered.

>> and in general
>> such projects seem to be aiming to move to wheels. So there aren't any
>> practical cases of "conda-only" packages.
>
> I'm not sure about "conda-only", but not pip-installable is all too common.

My benchmark is everything that used to distribute wininst installers
or binary eggs distributes wheels. I'd *like* to go beyond that, but
we haven't got the resources to help projects that never offered
binaries. Asking (and assisting) people to move from the old binary
standards to the new one should be a reasonable goal, though.

(I appreciate this is a less strict criterion than my previous
comments about "losing access to the scientific stack" could be taken
as implying. My apologies - some of my earlier comments probably
included a bit too much rhetoric :-))

>> > And for even that to work, we need a way for everything installable by
>> > pip
>> > to be installable within that conda environment -- which we could
>> > probably
>> > achieve.
>>
>> See above - without some serious resource on the conda side, that
>> "everything" is unlikely.
>
> conda can provide a full, pretty standard, python environment -- why do you
> think "everything" is unlikely?

Because the pip/wheel ecosystem relies on projects providing their own
wheels. Conda relies on the "conda guys" building packages. It's the
same reason that a PyPI build farm is still only a theory :-) Or were
you assuming that package authors would start providing conda
packages?

Paul