On Wed, Mar 27, 2013 at 10:09 PM, Donald Stufft <donald@stufft.io> wrote:
Right, which is why I think the ability to install from a raw source is a good feature for an installer, but not for the dependency metadata.
Sure - I never said the dependency metadata had to be able to say *where* you get the raw source from, just that the tool for resolving dependencies needed to be able to process raw source into something installable.
Following that we just need a standard way for a raw source tarball to declare what it's builder is, either via some sort of file that tells you that, or a build script , or something along those lines.
Yep. Static configuration is a *must*, here, though, as we want to move away from arbitrary setup script writing by package authors: in general they are really bad at it. A lot of setuptools' odd build-time features (like sandboxing) exist specifically because people write whatever crap they want in setup.py and have zero idea how to actually use/integrate with distutils. One interesting feature that would be possible under a configuration-based system is that you could actually have an installer with a whitelist or blacklist for build tools and setup-requires, in order to prevent or limit untrusted code execution by the overall build system. This would make it slightly more practical to have, say, servers that build wheels, such that only tools the servers' owners know won't import or run arbitrary code are allowed to do the compiling. (Not that that should be the only security involved, but it'd be a cool first-tier sanity check.) (Interestingly, this is also an argument for having a separate "tests-require-dist" in metadata 2.0, since testing tools *have* to run arbitrary code from the package, but archivers and builders do not.) Nick wrote:
However, as I've said elsewhere, for metadata 2.0, I *do not* plan to migrate the archiving or build steps away from setup.py. So "give me an sdist" will be spelled "python setup.py sdist" and "give me a wheel file" will be spelled "python setup.py bdist_wheel".
Works for me. Well, sort of. In principle, it means you can grow next generation build systems that use a dummy setup.py. In practice, it means you're still gonna be relying on setuptools. (Presumably 0.7 post-merge, w/bdist_wheel support baked in.) At some point, there has to be a new way to do it, because the pain of creating a functional dummy setup.py is a really high barrier to entry for a build tool to meet, until all the current tools that run setup.py files go away. IMO it'd be better to standardize this bit *now*, so that it'd be practical to start shipping projects without a setup.py, or perhaps make a "one dummy setup.py to rule them all" implementation that delegates everything to the new build interface. I can certainly understand that there are more urgent priorities in the short run; I just hope that a standard for this part lands concurrent with, say, PEP 439 and distlib ending up in the stdlib, so we don't have to wait another couple years to begin phasing out setuptools/distutils as the only build game in town. I mean, it basically amounts to defining some parameters to programmatically call a pair of sdist() and bdist_wheel() functions with, and a configuration syntax to say what distributions and modules to import those functions from. So it's not like it's going to be a huge time drain. (Maybe not even as much has already been consumed by this thread so far. ;-) ) Nick also wrote:
There's also an interesting migration problem for pre-2.0 sdists, where we can't assume that "python setup.py bdist_wheel && pip install <created wheel>" is equivalent to "python setup.py install": projects like Twisted that run a post-install hook won't install properly if you build a wheel first, since the existing post-install hook won't run.
It's an interesting problem, but one where my near term plans amount to "document the status quo".
Yeah, it's already broken and the new world order isn't going to break it any further. Same goes for allowing pip to convert eggs; the ones that don't work right due to bad platform tags, etc. *already* don't work, so documenting the status quo as a transitional measure is sufficient. Heck, in general, supporting backward compatible stuff that suffers from the same problems as the stuff it's being backward compatible with is a no-brainer if it lets people get on the new so we can phase out the old. (Which is why I love that Vinay is looking into how to make wheels more usable for some of eggs less-frequent but still important use cases: it makes it that much easier to tell someone they don't need to stay on setuptools to do the same stuff.)