[Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Sun, May 8, 2016 at 5:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> > any python developer is going to
> > run into these issues eventually...
>
> Aye, I know - conda was one of the systems I evaluated as a possible
> general purpose tool for user level package management in Fedora and
> derivatives (see
>
> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement#Directions_to_be_Explored
> for details).
>
> Hence my comment about conda not currently solving *system integrator*
> problems in the general case - it's fine as a platform for running
> software *on top of* Fedora and for DIY integration within an
> organisation, but as a tool for helping to *build Fedora*, it turned
> out to introduce a whole slew of new problems


Sure -- I don't think it was ever intended to support rpm et. all -- it's
to be used INSTEAD of rpm et. all -- for the most part, rpm has solved the
problems that conda is trying to solve -- but only for rpm-based Linux
systems. And I'm going to guess that Continuum didn't want to:

build packages for rpm systems (which ones?)
build packages for deb-based systems (which ones?)
build packages for gentoo
build packages for arch..
.....
build packages for homebrew

build packages for cygwin
build packages for Windows.. OOPS, there IS not package manager for
Windows!!

And, AFAICT, none of those package management systems support
"environments", either.

Clearly -- a tool like conda was required to meet Continuum's goals -- and
those goals are very much the same as PyPa's goals, actually. (except the
curated collection of packages part, but conda itself is not about the
Curation...)

However, the kinds of enhancements we're now considering upstream in
> pip should improve things for conda users as well - just as some folks
> in Fedora are exploring the feasibility of automatically rebuilding
> the whole of PyPI as RPMs


yes -- that's a good analogy -- for python packages, conda relies entirely
on distutils/setuptools/pip -- so yes, making those tools better and more
flexible is great.

but I'm still confused about what "the kinds of enhancements we're now
considering upstream in pip" are.

Here are a few:

More flexibility about the build system used
Easier to get metadata without actually initiating a build

These are great!

But I started this whole line of conversation because it seemed that there
was desire for:

Ability to isolate the build environment.
Ability to better handle/manage non-python dependencies

These are what I was referring to as mission-creep, and that overlap with
conda (and other tools).


> (twice, actually, anchored on
> /usr/bin/python and /usr/bin/python3), it should eventually be
> feasible to have the upstream->conda pipeline fully automated as well.


yeah -- there's been talk for ages of automatically building conda packages
(on the fly, maybe) from PyPi packages. But currently on conda-forge we've
decided to NOT try to do that -- it's turned out in practice that enough
pypi packages end up needing some hand-tweaking to build. So teh planned
workflow is now:

Auto-build a conda build script for a PyPi package
Test it
Tweak it as required
Add it to conda-forge.

Then -- *maybe* write a tool that auto-updates the PyPi based packages in a
chron job or whatever.

So not quite a automated conda-PyPi bridge, but not a bad start.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160509/55d93f13/attachment-0001.html>