[Distutils] Future of setuptools and buildout
Wes Turner
wes.turner at gmail.com
Fri Aug 19 01:17:19 EDT 2016
On Thu, Aug 18, 2016 at 4:54 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Wed, Aug 17, 2016 at 6:45 PM, Daniel Holth <dholth at gmail.com> wrote:
>
>> And a while back I argued against setuptools-lite, because I thought it
>> would not solve the poor extensibility problem that stems from its basic
>> distutils derived design... which includes all the classes and subclasses
>> that have to work together to do its thing. My own thinking is that
>> setuptools is fine, except that when you need to extend it you have a big
>> problem; so make it easy to replace.
>>
>
> I never had any fantasies that setuptool-lite would improve on this in a
> way shape or form -- indeed, a better build system is likely to need to be
> built from scratch, or on a totally different framework, anyway.
>
>
>> Does pip monkey patch setuptools? I thought it just forced setuptools
>> even if a package only asked for distutils.
>>
>
> good point -- not really a monkey patch, but it does force particular
> configuration.
>
>
>> So you are suggesting setuptools_lite would not honor the setup_requires
>> = [] and test_requires = []
>>
>
> essentially, yes. Though It would be nice if that data could be available
> for pip -- and I"m not sure how that is done at this point.
>
>
>> and perhaps "setup.py install" would also throw an error?
>>
>
> Probably not -- but it could only install from local source -- like
> distutils. Though if we could re-work it to only do develop mode and build
> wheels, that may be good -- you'd need pip to install it.
>
>
>> What else? I would suggest that to effectively argue the idea requires
>> reading the source code and enumerating exactly which complexity goes away
>> with the _lite version.
>>
>
> I don't know that reading the source is the issue -- I'm thinking entirely
> in terms of stripping out the API, so there may be things that need to stay
> in, but we don't expose them. though maybe even that's harder than I think.
>
> But yes -- enumerating what would and wouldn't be supported would be a
> good start. But here is the guideline:
>
> It seems we have a (maybe vague) vision of what we want the tool ecosystem
> to look like that involves separation of concerns, such that each component
> is replaceable without having to change the others:
>
> The concerns:
> - building a package
> - installing a package
> - dependency management
> - run-time version management
> - plugin handling, etc.
> - resource managment
>
> setuptools does all of these, yes? but think of these in terms of when
> they come into play:
>
> build time:
> - building a package
>
- building c extensions
- building NumPy (fortran,)
> install time: installing a package;
> - installing a package
> - dependency management
>
run time
> - run-time version management
> - plugin handling, etc.
>
- running tests
- (It's much faster/easier/consistent/reproducible to run tests when all
of the dependencies are bundled into one executable ZIP (e.g. PEX))
>
> so we have:
>
> - The mythical build tool
>
- TBH, I don't know much about pyproject.toml
- "PEP 518 -- Specifying Minimum Build System Requirements for Python
Projects"
https://www.python.org/dev/peps/pep-0518/
And then there are a number of build tools which work with multiple
languages (because just solving for Python doesn't get it):
- Conda (Python)
- meta.yaml, build.sh, build.bat *
- http://conda.pydata.org/docs/build_tutorials.html
-
http://conda.pydata.org/docs/build_tutorials/pkgs.html#conda-build-skeleton
(pypkg -> conda)
- http://conda.pydata.org/docs/building/meta-yaml.html#build-section
- http://conda.pydata.org/docs/bdist_conda.html (condapkg -> pypkg)
- Blaze (Java, Python?; Google) -> Open Sourced as Bazel
- Pants (Python; Twitter):
- BUILD files
- https://pantsbuild.github.io/python_readme.html
- https://pantsbuild.github.io/3rdparty_py.html (pip requirements.txt)
- https://pantsbuild.github.io/scala.html
- https://pantsbuild.github.io/go_readme.html
- Buck (Java, Facebook):
- https://buckbuild.com/rule/prebuilt_python_library.html (.egg)
- https://buckbuild.com/rule/python_binary.html
- https://buckbuild.com/rule/python_library.html
- https://buckbuild.com/rule/python_test.html
- https://buckbuild.com/rule/cxx_library.html
- https://buckbuild.com/rule/go_library.html
- https://buckbuild.com/rule/rust_library.html
- Bazel (Java, Python; Google)
- http://bazel.io/docs/be/python.html
- http://bazel.io/docs/be/python.html#py_binary
- http://bazel.io/docs/be/python.html#py_library
- http://bazel.io/docs/be/python.html#py_test
- https://bazel.io/docs/be/c-cpp.html
- https://github.com/bazelbuild/rules_go
... PEX is based on PEP 441:
- "PEP 441 -- Improving Python ZIP Application Support"
https://www.python.org/dev/peps/pep-0441/
- https://github.com/pantsbuild/pex/blob/master/docs/whatispex.rst
- Again, wheel includes a file manifest in the .whl .zip with SHA256
checksums:
https://bitbucket.org/pypa/wheel/src/cf4e2d98ecb1/wscript#wscript-69
- AFAIU, if you specify different install paths, there's no way to do
something like **debsums** / **rpm -V**
(which fpm handles well)
> - pip
> - maybe some of the setuptools spun off -- like pkg_resources
>
> So why aren't we there yet?
>
> One reason is that setuptools the only thing out there that does the
> building and some of the run-time stuff. So people use it. And now you
> can't replace any of it without making a mess.
>
> However, pip_wheel have indeed addressed much of the install-time and
> dependency management issues. So folks use them, and they work well.
>
> So what problem do I want setuptool-lite to solve?
>
> I actually have spent a lot of time using conda to do my package
> installing and dependency management. So I want a way to build a python
> package that JUST builds the package.
>
> In practice, I've found (most) of the flags and work arounds to diable
> etuptools "features", but really?
>
> python setup.py install --single-version-externally-managed
>
> Oh right, that's not enough, I need a record file:
>
> setup.py install --single-version-externally-managed --record record.txt
>
> Then I want to install in develop mode:
>
> python setup.py develop
>
> OOPS! now it's easy installing a bunch of dependencies! arrgg!
>
> OK, -- no-deps it is.
>
> Google around, this kind of thing is a pain in a lot of people's butts
> --even though it call all be worked around.
>
> So my vision of setuptool-lite is that is simply does not do the things
> that vision of a build tool should not do.
>
> i.e.:
>
> never easy-install anything!
> don't try to resolve dependencies
>
> others??
>
> One "lite" version of setuptools that I personally think would be cool
>> would be pkg_resources as a separate package on pypi. Setuptools would
>> require it as a dependency.
>>
>
> Yes, I'd love to disable pkg_resources too, but couldn't until it was spun
> off.
>
> In fact, years ago, long before pip, setuptools drove me crazy with how it
> combined run-time, build time and intall time stuff. A serious problem when
> I wanted to bundle something up with py2app (or py2exe, or...).
>
> So what would be accomplished?
>
> package builders could simply put:
>
> from setuptools_lite import setup
>
> (etc.)
>
> and then their user would never accidentally easy install anything, etc.
> And it would serve as documentation / notification if a user was using a
> what-should-be-deprecated feature.
>
> By the way, this is totally acknowledged to be a hacky, temporary mess
> that might only help a bit to get us closer from being locked into
> setuptools.
>
> (though maybe it would be a place to work through some of the issues with
> a build API)
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/cf163238/attachment-0001.html>
More information about the Distutils-SIG
mailing list