[Distutils] moving things forward (was: wheel including files it shouldn't)

Fri May 6 17:49:56 EDT 2016

On Fri, May 6, 2016 at 9:39 AM, Donald Stufft <donald at stufft.io> wrote:

> On May 6, 2016, at 11:54 AM, Chris Barker <chris.barker at noaa.gov> wrote:
>
> So my point is about scope-creep -- if you want the PyPa tools to solve
> all these problems, then you are re-inventing conda -- better to simply
> adopt conda (or fork it and fix issues that I'm sure are there….)
>
>
> Adopting Conda is unlikely to ever happen for the defaulting tooling.
>

I didn't put the emphasis right in that sentence -- I was not actually
suggesting that conda be adopted -- what I was suggesting was the we DON'T
expand the scope of pip et al to be able to do everything that conda does.

That being said, I think there are some misunderstanding here that may be
relevant to this discussion:

> The problems that the default tooling attempt to solve are significantly
> harder than the problems that Conda attempts to solve
>

kinda-sorta -- conda is a packaging system. period, end of story (actually
it is an isolated environment system, too -- is that packaging? )-- it is
NOT a build system. Right now, you can invoke pip to:

* Find and install a binary package (wheel) -- this conda does
* Find, download, build and install install a source package -- nope, conda
doesn't do anything like that.
* Build and install a local package from source
* Install a local package from source in "develop mode" (editable mde) --
actually conda does have that -- though I'm not sure who well it works, of
it it's python specific.

But the fact that this all is (apparently) done by one tool is actually a
problem -- we have a tangled mess of binary and source and building and
installing, and it isn't obvious to the user what they are getting ) often
they don't care -- as long as it works :-) ) And, under the hood, it's all
driven by setuptools, which also has duplicate and slightly different
functionality that can be accidentally invoked (fun with easy_install!).

So I thought the goal now was to clean up an untangle this mess, yes?

and switching to it would be a regression.
>

only in the sense the switching to pip (and getting rid of setuptools)
would be a regression....

> The primary benefit of Conda’s packages over Wheels are that they have a
> curated repository with people who are ensuring that things build correctly
> and since they don’t rely on authors to do it correctly, they don’t have to
> wait for, or convince authors to do this fresh new thing.
>

That simply is not true. Let's not mix social issues from technical ones.
Continuum set out to create a curated set of packages, yes -- but they
didn't put the enormous amount of work into conda for no reason -- they did
it because pip + wheel didn't meet their needs (and it doesn't meet a lot
of our needs, either). And, as I understand it, they came to the Python
community, and said something along the lines of "the packaging tooling
doesn't meet the needs of the scientific community, can we contribute
changes, improvements" -- and were told something along the lines of: "We
don't want the standard tooling to solve those problems -- go make what you
need, and then maybe we'll talk"

(I can dig out the message from Travis Oliphant about this if you like)

And they did go out and make something else, and they have released it as
an open source project, and a number of folks are adopting it to solve
their problems, too.

> The problem is, none of those benefits are something that would apply if
> we decided to throw away the 588,074 files that are currently able to be
> installed on PyPI.
>

conda aside, I think there is a real hang-up here. yes, of course you don't
want to throw away all that -- we absolutely need to maintain the current
packaging, building structure to support existing projects. But that
doesn't mean something new and better can't be built to be used in parallel
-- as folks move forward with pacakge development, they can adopt the new
system or they can keep using teh same old setup.py they have. MAybe
decades later something will get depreciated. But this approach is what got
us in the ugly mess to begin with.

Us deciding that Conda is the thing to use isn’t going to magically create
> an army of volunteers to go through and take all 80,000 packages on PyPI
> and ensure that we get a correctly compiled package on every platform for
> each version. If we could do that we could just convince everyone to go out
> and build binary packages, we could just do that with Wheels without
> requiring forklifting an entire ecosystem.
>

Again, conda packages and wheels are not the same thing -- there are
technical differences, and those differences really are why continuum uses
it for Anaconda, and why a lot of folks are adopting it for community led
efforts, like conda-forge. And there are a LOT of packages on conda-forge
despite it being very new and only a small handful of people contributing.

https://conda-forge.github.io/feedstocks.html

> While wheels are optimized for the pure Python case, there is nothing
> preventing them from being used to install anything else (Erlang or R or
> even Python itself).
>

OK, wheel is essentially "just" a archive -- so are conda packages -- they
are the same in that regard. but the tools that build and install wheels
aren't set up to support non-python. (and virtualenv vs. conda
environments, etc)

> The pynativelib stuff is proof of the ability to do just that— and in
> distutils-sig land we tend to care a lot more about how these things will
> affect our downstream consumers like Debian and such than Conda needs to
> (or should need to!).
>

if you split out building from package management, then the package
management has no impact on Debian et al. They use their package
management, and python's build system -- isn't that how it's done now?

Now, this isn’t to say that Conda is bad or anything, but it’s use as a
> replacement for the current ecosystem is about as interesting as suggesting
> we all adopt RPM, or apt-get, or Chocolatey, or Ports, or Pacman or
> whatever flavor of downstream you wish to insert here.
>

Again -- not suggesting that -- though I honestly think it's more driven by
"not invented here" than real technical reasons. A couple years ago, the
Python community very clearly said they didn't want to support non-python
libs, etc, etc. Now apparetnly we've changed our minds, but  want to keep
down the road we started out on...

Now I see scope creep pulling in -- we really want the "standard" python
packaging system to be able to support complex packages like numpy, scipy,
etc. So If the community has changed its mind, and wants to support that,
then it makes sense to step back a second and ask yourselves what the
architecture of the future system should look like? What do you want to
support? If nothing else, there have got to be lessons that can be learned
from conda (and of course, rpm, deb etc, too)

BTW, if PyPa does eventually get around to re-inventing conda -- great!

I was pretty unhappy when conda came out -- I was putting in real efforts
to try to get wheels built for various things on OS-X, and looking at how
to support my users with pip, pypi, and wheel.

But in the end, it turned out that conda made a whole lot of this a whole
lot easier - it really did (some of that technical, some of it social, but
certainly mostly technical). So I've been putting efforts into using conda
for my stuff, and supporting the community when I can.

But, of course, it turns out not all of my users want to use conda, so now
I need to put efforts into building wheels of my stuff, too -- and I am NOT
looking forward to that. So one system to rule them all would be great.

So what is the scope of all this?

Do you want to support non-python C libs as first class citizens? (and
Fortran, and...)

Do you want to support isolated environments (for not just python)

Do you want to support non-python systems as well -- R, perl, Julia, who
knows?

(I'm sure there are others....)

If that's the long view, great -- but start thinking about it now.

OK -- didn't intend that to be so long!

Carry on, I'll shut up now.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160506/5c4d700e/attachment-0001.html>