[Distutils] Future of setuptools and buildout

[Chris Barker]

On Thu, Aug 18, 2016 at 10:17 PM, Wes Turner <wes.turner at gmail.com> wrote:

> setuptools does all of these, yes? but think of these in terms of when
>> they come into play:
>>
>> build time:
>>    - building a package
>>
>
> - building c extensions
> - building NumPy (fortran,)
>

exactly! which is WHY we want/need more flexibility in build tools --
distutils is a mess to extend to do other things. (though is does do a fine
job with pure C extensions, which is what it was designed for).



> install time: installing a package;
>>   - installing a package
>>   - dependency management
>>
> run time
>>   - run-time version management
>>   - plugin handling, etc.
>>
>
> - running tests
>   - (It's much faster/easier/consistent/reproducible to run tests when
> all of the dependencies are bundled into one executable ZIP (e.g. PEX))
>

hmm - it seems most of us use a test-running for this -- so I put that in a
separate category. But you need the package management tool to be able to
easily set up an environment for testing -- i.e install the package and all
its dependencies, then run the tests. I don't know anything about PEX, but
I'd really NOT want the test runner to rely on any particular installation
setup.

And, as it happend, test running is one thing setuptools does not
currently, do so, we have pytest and nose and ??? that actually ARE
separate pluggable test running systems.


>
>> so we have:
>>
>>  - The mythical build tool
>>
>
> - TBH, I don't know much about pyproject.toml
>   - "PEP 518 -- Specifying Minimum Build System Requirements for Python
> Projects"
>      https://www.python.org/dev/peps/pep-0518/
>

pyproject.toml is NOT a build tool at all -- but it is a step towards being
able to support other build and packaging tools.

so maybe version 2 of setuptools_lite should rely on pyproject.toml.


> And then there are a number of build tools which work with multiple
> languages (because just solving for Python doesn't get it):
>
> - Conda (Python)
>

conda IS NOT a build tool -- I was pretty disappointed when I realized
that, but what can you do? cross-platform, cross-language build tools are a
really hard problem -- no one wants to write another one :-). And this is a
good thing -- what we WANT is for the packaging tool and the build tools to
be separate.


>   - meta.yaml, build.sh, build.bat *
>

the meta.yaml provides all the meta-data (kind like a higher level
pyproject.toml, and then simply invokes the build scripts. but you need to
write those build scripts -- conda provides literally no help with that.
most of them simply invoke whatever build system the library you are trying
to [package already uses -- setuptools, make, cmake, ....

However, since you brought up testing earlier -- conda does have a nice
system for testing -- it isn't a test runner, but it does provide isolated
environments (a job of the installer???) -- so conda-build can:

setup an environment for building
build the package
Then:

setup an environment for testing
installed the package that was just built
invoke a test-runner

This is REALLY nice -- you can have all your test code run with the built
package, with the environment that you have specified, on the platform you
have built for (on).

This has caught all sorts of stupid errors for me -- forgetting a
dependency, etc...

Whether that should be built into the packaging tool I suppose is
debateable. But maybe it's not -- conda-build is a separate tool -- it
relies on the conda system, but it does it's own thing. conda packages are
fairly simple archives -- it's very possible to create them with other
tools.

- Blaze (Java, Python?; Google) -> Open Sourced as Bazel
>
- Pants (Python; Twitter):
> - Buck (Java, Facebook):
> - Bazel (Java, Python; Google)
>

Maybe the alternative to setuptools_lite is to set up one of these other
tools to work well (easily) for building python packages. If it can do
everything that setuptools can do (that we want setuptools to do), and just
as easily, maybe that's the next step forward.

but if that isn't happening soon, then a setuptools_lite would be a useful
step forward.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/52444dca/attachment.html>