[Distutils] moving things forward (was: wheel including files it shouldn't)

Sat May 7 14:15:50 EDT 2016

On Sat, May 7, 2016 at 6:51 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 7 May 2016 01:55, "Chris Barker" <chris.barker at noaa.gov> wrote:
> > So my point is about scope-creep -- if you want the PyPa tools to solve
> all these problems, then you are re-inventing conda -- better to simply
> adopt conda (or fork it and fix issues that I'm sure are there....)
>
> conda doesn't solve these problems either - it solves the *end user*
> problem for data analysts (install the Python library they want to use),
>
I really need to make this clear -- there is NOTHING "data analyst"
specific about these problems -- they do come up more in the computational
programming domain, but there are any number of other application that have
the same problems (pyQT has come up in this conversation, yes?) -- and
we're finding conda to be a good solution for our web development needs,
too -- a conda environment  is kinda like a lighter-weight, platform
independent docker container. And of course, there is more an more data
analysis going on behind web services these days too -- any python
developer is going to run into these issues eventually...

> by ignoring the system integrator one (interoperate with the system
> integrator's existing platform level package management systems, of which
> we all tend to have our own with no plans for consolidation)
>
> That's by design, though - conda was specifically created as a language
> independent cross-platform platform, not as a cross-platform plugin
> management system for Python runtimes.
>
exactly!

> For a long time I was convinced the existence of conda and Linux
> containers as end user tools meant we wouldn't ever need to solve these
> problems at the language runtime layer, but it's since become clear to me
> that there's significant value in solving these problems in a way that
> doesn't care about how you obtained your Python runtime.
>
yup -- that would be the scope creep I'm referring too :-)

But now I'm confused about what problems we're talking about. From above:

"""
interoperate with the system integrator's existing platform level package
management systems
"""

you mean rpm, .deb, homebrew, etc?  distutils did have (does it still)
bdist_rpm (and, for that matter bdist_wininst) -- which I see as an attempt
to inter-operate with the system integrator's platform. But I think it
turns out that this is a hopeless task -- there are just too many different
systems to consider and maintain -- much better to let the external systems
handle that. And it got really ugly when you wanted to mingle virtualenv
with rpm, etc....

The "solution" to that is to actually not do it -- use Docker or Conda if
you want an isolated environment.

What all this means to me is that  we really do need to keep the building
separate from the packaging -- so that the building tools can be used /
supported by the systems integrator. But people shouldn't be using rpm to
manage their system, and expect to install binary wheels...

The trick is that the pip/wheel system is really handy -- certainly for
Windows and OS-X that don't provide a standard system package manager. It
is nice to be able to "just pip install" various packages. But the problem
is that it is a half way solution -- it only works well for python-centered
packages -- pure python is trivial of course, but custom written C
extensions work great too -- but as soon as you get external dependencies,
it all goes to heck. We can improve that -- defining the manylinux platform
is a great step, but to really get there, there needs to be a way to handle
third party libs (which we could probably do with the existing tools by
establishing conventions), but then you get to non-python support tools:
Fortran compilers, other language run times, all sorts of stuff. Oh, and
then we want isolated environments, and they need to isolate these other
non-python tools, so ....

I think it's WAY beyond the current PEP under discussion, but we really
should map out a "where we'd like to get" plan.

"significant value in solving these problems in a way that doesn't care
about how you obtained your Python runtime."

I don't get this -- I don't think it's possible to not care about the
python run-time -- we've only gotten this far because most folks have more
or less standardized on the python.org builds as the standard python build.
Even when you get your python elsewhere (Anaconda, for instance) they take
pains to keep it compatible.

I think we should count on python itself being managed by the same package
manager as the packages -- or at least conforming to a standard (which I
think we have with teh ABI tags, yes?)

BTW, I only recently learned that there are more "other" python builds out
there than I thought. Apparently the Visual Effect folks are on the habit
of building their own python2.7 on Windows, so they can use a newer MS
compiler, and then build their extensions to be compatible (all so you can
use modern C++ in the extension code). I don't know that they are careful
about setting the ABI tags, but they could be.

> For a long time I was convinced the existence of conda and Linux
containers as end user
> tools meant we wouldn't ever need to solve these problems at the language
runtime layer

to an extent that is true -- but it does mean a proliferation of options,
and as I think you pointed out earlier, that sucks because you need someone
to support all those options -- for instance, most conda packages are built
by third parties now - not the original package author. This is (very
slowly) starting to change (Ned Batchelder recently asked about building
conda packages for coverage...). and of course, the same is true with
homebrew, and rpm, and deb, and....

So yeah, it would be nice to have ONE thing that python package developers
need to do to make their work available to everyone.

So I'll leave off with this:

If there is even going to be one way for python package developers to
provide binary packages for their users -- that system will need to support
all (most) of what conda supports. Every feature in conda was added because
it was needed.

And THAT's what I've been meaning every time i say "reimpliment conda"

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160507/cc02af4d/attachment-0001.html>