On Mon, Mar 25, 2013 at 7:15 PM, PJ Eby <pje@telecommunity.com> wrote:
On Mon, Mar 25, 2013 at 5:08 PM, Paul Moore <p.f.moore@gmail.com> wrote:
There's a longer-term issue that occurred to me when thinking about pip's role as a "builder" or an "installer" (to use Nick's terminology).
As I understand Nick's vision for the future, installers (like pip) will locate built wheels and download and install them, and builders (like distutils and bento) will be responsible for building wheels. But there's an intermediate role which shouldn't get forgotten in the transition - the role that pip currently handles with the "pip wheel" command. This is where I specify a list of distributions, and pip locates sdists, downloads them, checks dependencies, and ultimately builds all of the wheels. I'm not sure whether the current idea of builders includes this "locate, download and resolve dependencies" function (distutils and bento certainly don't have that capability).
Yes, and to make things even more interesting, consider the cases where there are build-time dependencies. ;-)
I would guess that installing from sdists (and revision control) is probably here to stay, along with the inherent coupling between "build" and "fetch" functions.
Right now, the "build" side of setuptools fetches build-time dependencies, but in the New World, ISTM that a top-level build tool would just be something that reads the package metadata, finds build-time dependencies, and then runs some entry points to ask for a wheel to be spit out (or to get back a data structure describing the wheel, anyway).
This part could be standardized and indeed could be just something that pip does.
Another piece that hasn't seemed well-specified so far is what Nick called "archiving" - creating the sdist. Or perhaps more precisely, generating an sdist PKG-INFO and putting the stuff together.
IMO, a good install tool needs to be able to run the archiving and building steps as well as installing directly from wheels. However, although metadata 2.0 provides us with a good basis for running build and install steps, there really isn't anything yet in the way of a standard for sdist generation.
Of course, we have "setup.py sdist", and the previous work on setup.cfg by the distutil2 team. We could either build on setup.cfg, or perhaps start over with a simple spec to say how the package is to be built: i.e., just a simple set of entry points saying what archiver, builder, etc. are used. Such a file wouldn't change much over the life of the package, and would avoid the need for all the dynamic hooks provided by distutils2's setup.cfg.
In the degenerate case, I suppose, it could just be "pyarchiver.cfg" and contain a couple lines saying what tool is used to generate PKG-INFO.
On the other hand, we could draw the line at saying, pip only ever installs from sdists, no source checkouts or tarballs. I'm not sure that's a reasonable limitation, though.
On the *other* other hand, Perhaps it would be best to just use the setup.cfg work, updated to handle the full metadata 2.0 spec. As I recall, the setup.cfg format handled a ridiculously large number of use cases in a very static format, and IIRC still included the possibility for dynamic hooks to affect the metadata-generation and archive content selection processes, let alone the build and other stages.
But then, is that biasing against e.g. bento.info? Argh. Packaging is hard, let's go shopping. ;-)
On balance, I think I lean towards just having a simple way to specify your chosen archiver, so that installing from source checkouts and dumps is possible. I just find it annoying that you have to have *two* files in your checkout, one to say what tool you're using, and another one to configure it.
(What'd be nice is if you could just somehow detect files like bento.info and setup.cfg and thereby detect what archiver to use. But that would have limited extensibility unless there was a standard naming convention for the files, or a standardized format for at least the first line in the file or something like that, so you could identify the needed tool.)
The problem we are solving first is not "setuptools", it is simply that right now Python programs are usually installed by downloading and then running a program that performs the actual install. Instead, we're going to let the installer do the actual install. The funny thing is that all the same things happen. We just move the responsibilities around a little bit, things like Bento or distutils2 don't have to implement their own installer code, and we don't have to worry about that code doing bad things like messing up our systems or accessing the Internet. The installer might do new clever things now that it really controls the install. There are a tremendous number of things you can do from there, mostly undeveloped, including decoupling the rest of the packaging pipeline all the way down to the humble sdist, but we don't really need to change the pip user interface. All the same things will continue to happen, just rearranged in a more modular way. The MEBS "ultimate punt" design is that you would iterate over build plugins to recognize an sdist. The sdist would be defined as anything recognized by a plugin. It's probably more practical to at least name the preferred build system in a very minimal setup.cfg. This is still a little bit ugly. In a normal "new style" sdist, you may be able to trust the PKG-INFO file, but when building from source control you can't, would need to inspect setup.cfg, and ask the build system to refresh the metadata.