This is not a hole. Historically there has been VERY STRONG resistance to any practical, incremental improvements to packaging. Instead we are expected to come up with something that is ideal and fully formed. The results is that packaging is not often improved. The proposal is just a way to make setup.py work a little better. We are going to have setup.py forever, but we hope it will not be the primary way to make new packages. We are going to have lists of requirements forever. We are going to need setup & build requirements specifications that actually work. Allowing a list of requirements in setup.cfg or a text file is a very simple way to move in the right direction. On Thu, May 15, 2014 at 1:35 PM, Donald Stufft wrote:

On May 15, 2014, at 1:21 PM, Daniel Holth wrote:

...

Glyph has suggested something that I've been wanting to do for a long time. "let me use setup_requires somehow so I can have abstractions in setup.py".

setup.py allows you to pass setup_requires = [] to the setup() call. Unfortunately, since setup.py needs setup_requires to be installed before it can run, this feature is crap. It also has the unfortunate side effect of recursively calling easy_install and installing the listed packages even when pip or no installer is being used.

Instead, I'd like to allow a list of requirements in setup.cfg:

[a section name] setup-requires = somepackage > 4.0 anotherpackage >= 3.2.1

As an alternative we could look for a file setup-requires.txt which would be the same as a pip-format requirements file.

I prefer doing it in setup.cfg because people are used to that file, and because it only accepts package names and not general pip or easy_install command line arguments.

This would be simple and a tremendous boon to anyone considering implementing a setup.py-emulating distutils replacement, or to anyone who just likes abstractions.

Metadata 2.0 is not a solution for this problem. It's late, it's more complicated, and for any "legacy" packages setup.py is the thing that generates metadata.json - not something that can only run if you parse a skeletal metadata.json, do what it says, and then overwrite metadata.json when dist_info is called.

Daniel

Even though it would bandaid over some pain points I'm against shoehorning this into the current specs. It's something we'll have to support *forever*. I think it makes sense to take the time to do it right instead of trying to toss more hacks and bandaids ontop of all of the existing hacks and bandaids.

The first law of holes says "If you find yourself in a hole, stop digging."

We're in a hole, so please, let's stop digging.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Come to think of it it would be pretty easy to implement this feature without involving pip at all by writing a small setup.py replacement that knows how to look for and install the aforementioned requirements. If it does not finds them then it shells out to pip to install them in a build-dep directory. It adds build-dep to sys.path, and runs real-setup.py in its own process.

since setup.py needs setup_requires to be installed before it can run,

this technique would be a great topic for the PPUG please post if you write it up. this feature is crap well, you can write command extensions that depend on "setup_requires" projects. what's an example you're thinking of, for one of these "abstractions"?

I'm of course hoping that someone uses the feature to do a setup.py-command-line-interface-compatible distutils replacement (rather than a distutils extension) like what Bento does with its pip-compatible setup.py replacement.

ah, I see. interesting. I admit my initial reaction to "quack like a setup.py" alternative build systems is that it sounds invasive and sneaky, and I want it to be explicit and overt, like it will be in metadata 2.X. but on the other hand, I hear you about wanting to see alternatives have some way to get off the ground now.

On Thu, May 15, 2014 at 6:31 PM, Daniel Holth wrote:

On Thu, May 15, 2014 at 6:04 PM, Marcus Smith wrote:

...

...

I'm of course hoping that someone uses the feature to do a setup.py-command-line-interface-compatible distutils replacement (rather than a distutils extension) like what Bento does with its pip-compatible setup.py replacement.

ah, I see. interesting. I admit my initial reaction to "quack like a setup.py" alternative build systems is that it sounds invasive and sneaky, and I want it to be explicit and overt, like it will be in metadata 2.X. but on the other hand, I hear you about wanting to see alternatives have some way to get off the ground now.

There's a bit of a chicken-egg problem obviously.

For the moment I am more interested in just giving people a usable setup_requires.

Just now catching up on this, but I'm strongly in favor of fixing this issue. It's a simple addition to setuptools. I actually am about to release the first version of a package called astropy_helpers, which basically is a bundle of all the build tools used by the Astropy project so that other related projects can take advantage of them as well. This has the same chicken-egg problem--you want to be able to use astropy_helpers in setup.py, but you can't get to it without first calling setup(setup_requires=['astropy_helpers']). This can actually be gotten around by simply creating a dummy Distribution object and calling it like: from setuptools.dist import Distribution Distribution({'setup_requires': ['astropy_helpers']}) But that's an ugly hack. To that end I added bootstrap script that's imported at the top of setup.py that knows how to do this (it also has code for finding astropy_helpers as a git submodule if the project has one in its repo--useful for development). d2to1 takes a slightly different approach in that one doesn't pass anything to the normal setup() except for setup_requires=['d2to1']. Once d2to1 is bootstrapped it takes over the entire setup process, including reading additional setup_requires from setup.cfg as Daniel suggested. I'm actually planning on relaunching d2to1 with a new name (less tied to the defunct distutils2) because I still think it's a useful alternative to setup.py, while still working within the existing framework (and easily transferable to any new framework). Erik

On May 15, 2014, at 1:55 PM, Daniel Holth wrote:

This is not a hole.

Historically there has been VERY STRONG resistance to any practical, incremental improvements to packaging. Instead we are expected to come up with something that is ideal and fully formed. The results is that packaging is not often improved.

The proposal is just a way to make setup.py work a little better. We are going to have setup.py forever, but we hope it will not be the primary way to make new packages. We are going to have lists of requirements forever. We are going to need setup & build requirements specifications that actually work. Allowing a list of requirements in setup.cfg or a text file is a very simple way to move in the right direction.

I'm completely for incremental improvements *which move us closer towards the final end goal*. Hacking in this support is not that. It'll only reasonably work in pip 1.6+ and won't in older versions of pip nor easy_install, zc.buildout, or direct setup.py invocations. This represents an official recommended backwards compatibility break. It is much better to be able to work on either new formats or improvements to the existing formats which don't require changing the fundamental API contract of the old formats. Hacking it in by the project creating a fake setup.py that does the work is a much better solution since it doesn't break the API contracts and it doesn't represent another pseudo format that we'll have to support indefinitely. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA