Re: buildout and pip and wheel "Add support for installing wheels" https://github.com/buildout/buildout/issues/144 It's been awhile since I've worked with buildout (for Zope 2, Plone, AppEngine zipimports). This reads #egg= links from pip requirements files: - https://github.com/collective/collective.recipe.pip/blob/master/collective/r... This installs packages with buildout and pip (instead of zc.recipe.egg) into the eggs/ directory. IDK if it supports wheels? (it specifies --egg): - https://github.com/k4ml/mk.recipe.pip/blob/master/mk/recipe/pip/__init__.py ... this builds a usable merged namespace with symlinks to eggs: - https://github.com/collective/collective.recipe.omelette/ On Tuesday, August 16, 2016, Daniel Holth wrote:

There are two proposals to add pluggable build systems to Python sdists, and one of them will probably be implemented. You add a pyproject.toml to the root of your sdist, which the installer uses to install dependencies that setup.py itself needs to run. Second, you also tell pip which build system you are using, once you have done that, setup.py is no longer necessary - instead, pip could install and then invoke flit for example.

This isn't quite the situation you've outlined as setuptools will likely continue to have all its features. Instead, packages that don't use setuptools will also proliferate. It causes about the same problem for buildout without breaking older packages.

In this scenario buildout will need to be able to install wheels however it wishes to do so because that is the output of the pluggable build system. I don't have advice for you on the implementation, except perhaps it could call out to pip.

On Tue, Aug 16, 2016 at 12:59 PM Leonardo Rochael Almeida < leorochael@gmail.com javascript:_e(%7B%7D,'cvml','leorochael@gmail.com');> wrote:

...

Spinning this off into its own thread...

On 16 August 2016 at 13:10, Donald Stufft javascript:_e(%7B%7D,'cvml','donald@stufft.io');> wrote:

...

On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida < leorochael@gmail.com javascript:_e(%7B%7D,'cvml','leorochael@gmail.com');> wrote:

Specifically, buildout right now has setuptools as its only dependence, and buildout uses it to locate, download and build sdist dependencies, or directly download and use .egg dependencies, which are important for Windows platforms in the communities that use buildout and depend on compiled extensions.

Ah, I knew I was forgetting something. I think we should hold off on preventing egg uploads until setuptools can download and install wheels.

Which brings us to a question that I'm meaning to ask for a while.

It looks like we're close to removing all mentions of setuptools in pip. When this happens, it looks like pressure is going to start to mount on setuptools to drop the ability to install packages and limit itself on being just a build tool.

It makes sense that it should be so, considering that there would be no incentive to keep around two distinct implementations of how to locate, download and install stuff, one in pip (or whatever library pip uses for locating and installing packages) and another in setuptools.

In this future, I can imagine setuptools will simply complain loudly when the packages it requires to do its work are not installed yet, perhaps with a plugin mechanism for allowing another package to automatically resolve and installing packages, (like today setuptools-git extends setuptools to recognize files from git to be included in a package).

And people will promptly write a pip implementation of this plugin (or whatever it is that pip will use as a library to do it)... thereby inverting the dependency between setuptools and pip.

When this future arrives, buildout will be out of a mechanism for locating, downloading and installing packages (not counting old versions of setuptools, of course).

Questions:

1. Is the future outlined above considered likely?

2. If this future arrives, what package should buildout be ported to use? It should provide:

* The ability to locate packages and their versions, resolve their dependencies, and download them, whether they're sdists or wheels (or eggs, while we still have them) * Install each package separately into it's own directory that, if added to `sys.path`, allows this same package (or perhaps another one) to read information from entry points so that buildout can locate its own extensions and install console scripts.

Regards,

Leo _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org javascript:_e(%7B%7D,'cvml','Distutils-SIG@python.org'); https://mail.python.org/mailman/listinfo/distutils-sig

Which brings us to a question that I'm meaning to ask for a while.

It looks like we're close to removing all mentions of setuptools in pip. When this happens, it looks like pressure is going to start to mount on setuptools to drop the ability to install packages and limit itself on being just a build tool.

Yes, please! I argued a while back for a setuptools-lite -- it would do all the things setuptools does that we think it _should_ do, and not do any of the others -- like it wouldn't easy-install anything EVER. most likely it would be a fork of the setuptools code with a bunch of stuff disabled. Or maybe even a setting in setuptools itself: import setuptools setuptools.disable_legacy from setuptools import setup, find_packages, ... Why???? Folks rely on the various "features" of setuptools, some important ones like buildout. So it's going to be a long time before we can deprecate all that in setuptools itself. But we're trying for a future with better separation of concerns -- building, installing, packaging, run-time management. But as it stands now, people kind of HAVE to use setuptools to get desired build behaviour, but then their users an accidentally invoke features they don't want -- to teh point where pip goes in and monkey patches the darn thing. IF there were a setuptools_lite, user could simply do: import setuptools_lite as setuptools and they'd instantly get an error if they were using depreciated features, and their end users would never accidently easy install stuff :-) Anyway, this seem like a path forward, without having to wait for the future fabulous pluggable build system ..... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 18 August 2016 at 11:45, Daniel Holth wrote:

One "lite" version of setuptools that I personally think would be cool would be pkg_resources as a separate package on pypi. Setuptools would require it as a dependency.

I finally managed to relocate a discussion Jason, Donald and I had about this a couple of months ago: https://github.com/ncoghlan/pkg_resources_shim/issues/1 (The idea there was to make "pip install pkg_resources" just work by implicitly installing setuptools - at the moment running that command errors out, since Donald reserved the name on PyPI to prevent anyone using it as an attack vector against folks actually looking for setuptools) Creating a standalone "pkg_resources" project turns out to be relatively straightforward, but *removing* pkg_resources from setuptools turns out to create significant setuptools bootstrapping problems. That means the standalone project would need to use a different name, and any associated bugs would subsequently need to be fixed in two different projects, which seems like a bad idea. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Aug 18, 2016 at 4:54 PM, Chris Barker wrote:

On Wed, Aug 17, 2016 at 6:45 PM, Daniel Holth wrote:

...

And a while back I argued against setuptools-lite, because I thought it would not solve the poor extensibility problem that stems from its basic distutils derived design... which includes all the classes and subclasses that have to work together to do its thing. My own thinking is that setuptools is fine, except that when you need to extend it you have a big problem; so make it easy to replace.

I never had any fantasies that setuptool-lite would improve on this in a way shape or form -- indeed, a better build system is likely to need to be built from scratch, or on a totally different framework, anyway.

...

Does pip monkey patch setuptools? I thought it just forced setuptools even if a package only asked for distutils.

good point -- not really a monkey patch, but it does force particular configuration.

...

So you are suggesting setuptools_lite would not honor the setup_requires = [] and test_requires = []

essentially, yes. Though It would be nice if that data could be available for pip -- and I"m not sure how that is done at this point.

...

and perhaps "setup.py install" would also throw an error?

Probably not -- but it could only install from local source -- like distutils. Though if we could re-work it to only do develop mode and build wheels, that may be good -- you'd need pip to install it.

...

What else? I would suggest that to effectively argue the idea requires reading the source code and enumerating exactly which complexity goes away with the _lite version.

I don't know that reading the source is the issue -- I'm thinking entirely in terms of stripping out the API, so there may be things that need to stay in, but we don't expose them. though maybe even that's harder than I think.

But yes -- enumerating what would and wouldn't be supported would be a good start. But here is the guideline:

It seems we have a (maybe vague) vision of what we want the tool ecosystem to look like that involves separation of concerns, such that each component is replaceable without having to change the others:

The concerns: - building a package - installing a package - dependency management - run-time version management - plugin handling, etc. - resource managment

setuptools does all of these, yes? but think of these in terms of when they come into play:

build time: - building a package

- building c extensions - building NumPy (fortran,)

install time: installing a package; - installing a package - dependency management

run time

- run-time version management - plugin handling, etc.

- running tests - (It's much faster/easier/consistent/reproducible to run tests when all of the dependencies are bundled into one executable ZIP (e.g. PEX))

so we have:

- The mythical build tool

- TBH, I don't know much about pyproject.toml - "PEP 518 -- Specifying Minimum Build System Requirements for Python Projects" https://www.python.org/dev/peps/pep-0518/ And then there are a number of build tools which work with multiple languages (because just solving for Python doesn't get it): - Conda (Python) - meta.yaml, build.sh, build.bat * - http://conda.pydata.org/docs/build_tutorials.html - http://conda.pydata.org/docs/build_tutorials/pkgs.html#conda-build-skeleton (pypkg -> conda) - http://conda.pydata.org/docs/building/meta-yaml.html#build-section - http://conda.pydata.org/docs/bdist_conda.html (condapkg -> pypkg) - Blaze (Java, Python?; Google) -> Open Sourced as Bazel - Pants (Python; Twitter): - BUILD files - https://pantsbuild.github.io/python_readme.html - https://pantsbuild.github.io/3rdparty_py.html (pip requirements.txt) - https://pantsbuild.github.io/scala.html - https://pantsbuild.github.io/go_readme.html - Buck (Java, Facebook): - https://buckbuild.com/rule/prebuilt_python_library.html (.egg) - https://buckbuild.com/rule/python_binary.html - https://buckbuild.com/rule/python_library.html - https://buckbuild.com/rule/python_test.html - https://buckbuild.com/rule/cxx_library.html - https://buckbuild.com/rule/go_library.html - https://buckbuild.com/rule/rust_library.html - Bazel (Java, Python; Google) - http://bazel.io/docs/be/python.html - http://bazel.io/docs/be/python.html#py_binary - http://bazel.io/docs/be/python.html#py_library - http://bazel.io/docs/be/python.html#py_test - https://bazel.io/docs/be/c-cpp.html - https://github.com/bazelbuild/rules_go ... PEX is based on PEP 441: - "PEP 441 -- Improving Python ZIP Application Support" https://www.python.org/dev/peps/pep-0441/ - https://github.com/pantsbuild/pex/blob/master/docs/whatispex.rst - Again, wheel includes a file manifest in the .whl .zip with SHA256 checksums: https://bitbucket.org/pypa/wheel/src/cf4e2d98ecb1/wscript#wscript-69 - AFAIU, if you specify different install paths, there's no way to do something like **debsums** / **rpm -V** (which fpm handles well)

- pip - maybe some of the setuptools spun off -- like pkg_resources

So why aren't we there yet?

One reason is that setuptools the only thing out there that does the building and some of the run-time stuff. So people use it. And now you can't replace any of it without making a mess.

However, pip_wheel have indeed addressed much of the install-time and dependency management issues. So folks use them, and they work well.

So what problem do I want setuptool-lite to solve?

I actually have spent a lot of time using conda to do my package installing and dependency management. So I want a way to build a python package that JUST builds the package.

In practice, I've found (most) of the flags and work arounds to diable etuptools "features", but really?

python setup.py install --single-version-externally-managed

Oh right, that's not enough, I need a record file:

setup.py install --single-version-externally-managed --record record.txt

Then I want to install in develop mode:

python setup.py develop

OOPS! now it's easy installing a bunch of dependencies! arrgg!

OK, -- no-deps it is.

Google around, this kind of thing is a pain in a lot of people's butts --even though it call all be worked around.

So my vision of setuptool-lite is that is simply does not do the things that vision of a build tool should not do.

i.e.:

never easy-install anything! don't try to resolve dependencies

others??

One "lite" version of setuptools that I personally think would be cool

...

would be pkg_resources as a separate package on pypi. Setuptools would require it as a dependency.

Yes, I'd love to disable pkg_resources too, but couldn't until it was spun off.

In fact, years ago, long before pip, setuptools drove me crazy with how it combined run-time, build time and intall time stuff. A serious problem when I wanted to bundle something up with py2app (or py2exe, or...).

So what would be accomplished?

package builders could simply put:

from setuptools_lite import setup

(etc.)

and then their user would never accidentally easy install anything, etc. And it would serve as documentation / notification if a user was using a what-should-be-deprecated feature.

By the way, this is totally acknowledged to be a hacky, temporary mess that might only help a bit to get us closer from being locked into setuptools.

(though maybe it would be a place to work through some of the issues with a build API)

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 20 August 2016 at 02:46, Chris Barker wrote:

Maybe the alternative to setuptools_lite is to set up one of these other tools to work well (easily) for building python packages. If it can do everything that setuptools can do (that we want setuptools to do), and just as easily, maybe that's the next step forward.

but if that isn't happening soon, then a setuptools_lite would be a useful step forward.

Enabling folks to more easily use existing build systems like Scons, Waf, Meson, CMake, et al is one of the main goals of pyproject.toml. Daniel Holth has a working Scons proof of concept at https://pypi.python.org/pypi/enscons that actually works independently of pyproject.toml support in pip by using a boilerplate setup.py to do the bootstrapping and invocation of the build system. It's why we don't particularly want to replace distutils/setuptools with any build system in particular - the world actually has a surfeit of cross-platform build systems, and many of them are written in Python and are hence quite friendly to being bootstrapped for use in building Python extension modules. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 18 August 2016 at 08:18, Chris Barker wrote:

IF there were a setuptools_lite, user could simply do:

import setuptools_lite as setuptools

and they'd instantly get an error if they were using depreciated features, and their end users would never accidently easy install stuff :-)

Anyway, this seem like a path forward, without having to wait for the future fabulous pluggable build system .....

The problem with this is a pragmatic one rather than a philosophical one, in that setuptools is just plain *hard* to work on, and doesn't have a particularly robust test suite, so you can easily break other people's usage while trying to fix the particular issue you personally care about. Hence the ongoing efforts to let people more easily use tools that *aren't* setuptools, rather than attempting to disentangle and modularise setuptools itself - the architectural challenges are such that attempting to fix them would inevitably break existing usage, while we can get most of the same benefits more transparently through things like declarative dependencies for setup.py execution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia