On Fri, Jun 2, 2017 at 9:39 AM, Paul Moore p.f.moore@gmail.com wrote:
Note that there's an implication here - if we define the build process in terms of the effect of "going via a sdist", then we need to at least have an intuitive understanding of what that means in practice. I don't think it's a contentious point (even if the specific term "sdist" is open to debate), as I think repeatable builds are a well-understood idea. (It's at this point that the concerns of people who want incremental builds come in - we should support incremental builds in a way that preserves the "just like going via a sdist" principle. But again, they need to raise their concerns if they think we're missing something key to their use case).
So far my belief is that packages with expensive build processes are going to ignore you and implement, ship, document, and recommend the direct source-tree->wheel path for developer builds. You can force the make-a-wheel-from-a-directory-without-copying-and-then-install-it command have a name that doesn't start with "pip", but it's still going to exist and be used. Why wouldn't it? It's trivial to implement and it works, and I haven't heard any alternative proposals that have either of those properties. [1]
Relatedly, the idea of a copy_files hook doesn't make sense to me. The only reason pip wants to force builds through the sdist phase is because it doesn't trust the backend to make clean wheels, and it's willing to make its local directory builds much slower to get that guarantee. When you add copy_files, you lose that guarantee and you're still making local directory builds much slower, so what's the point? If the always-via-sdist plan doesn't work for either the simplest cases (flit) or the most complex (incremental builds), then is it really a good plan?
So it seems clear that we should do:
- add get_build_sdist_requires and build_sdist hooks to PEP 517 or a new PEP, whichever (and for clarity rename the current get_build_requires -> get_build_wheel_requires)
Beyond that, my preference is:
- 'pip install local-directory/' switches to building in-place
If the pip devs don't trust build systems in general, but (as suggested by copy_files discussion) are ok with trusting them if they promise to be super trustworthy, alternate proposal:
- add a 'in_place_build_safe = True' hook, which indicates that the build system has been carefully written so that this will generate the same result as building an sdist and then building that; pip checks for this to decide whether to build in place or to build an sdist first.
In principle this is a little silly (who doesn't think themselves trustworthy?), but it would let us continue to do build isolation for setuptools builds and any build system that hasn't put some thought into this, makes it clear where the responsibility lies if someone screws up, and backends that don't want to deal with building sdists from sdists could make this False for VCS checkouts and True for unpacked sdists.
If the pip devs are set on 'pip install local-directory/' always making some sort of copy, then I suggest:
- 'pip install local-directory/' makes an sdist and then builds it
- we also add something like 'pip devinstall local-directory/' that does an in-place build and then installs it, so that I don't have to make a separate 'devinstall' script and ship it separately
- 'flit' adds code to make sdist-from-sdist work. (One way: when building an sdist from a VCS checkout, make a list of all the ancillary files and include it in the resulting sdist. Or possibly just a list of all files + hashes. When asked to make an sdist from an arbitrary directory, check for this file, and if present use it as the list of ancillary files to include, and possibly check if any hashes have changed, and if so change the version number of the resulting sdist by appending "+dirty" or something; otherwise, use the current VCS-based system.)
One thing that's not clear to me: a crucial use case for sdists is (1) download, (2) unpack, (3) patch the source, possibly adding new files, (4) build and install. (After all, the whole reason we insist on distributing sdists is that open source software should be modifiable by the recipient.) Does flit currently support this, given the reliance on VCS metadata?
Other unresolved issues:
Donald had some concerns about get_wheel_metadata and they've led to several suggestions, none of which has made everyone go "oh yeah obviously that's the solution". To me this suggests we should go ahead and drop it from PEP 517 and add it back later if/when the need is more obvious. It's optional anyway, so adding it later doesn't hurt anything.
It sounds like there's some real question about how exactly a build frontend should handle the output from build_wheel; in particular, the PEP should say what happens if there are multiple files deposited into the output dir. My original idea when writing the PEP was that the build frontend would know the name/version of the wheel it was looking for, and so it would ignore any other files found in the output dir, which would be forward compatible with a future PEP allowing build_wheel to drop multiple wheels into the output dir (i.e., old pip's would just ignore them). It's clear from the discussion that this isn't how others were imagining it. Which is fine, I don't think this is a huge problem, but we should nail it down so we're not surprised later.
-n
[1] Donald's suggestion of silently caching intermediate files in some global cache dir is unreasonably difficult to implement in a user-friendly way – cache management is Hard, and I frankly I still don't think users will accept individual package's build systems leaving hundreds of megabytes of random gunk inside hidden directories. We could debate the details here, but basically, if this were a great idea to do by default, then surely one of cmake/autoconf/... would already do it? Also, my understanding is the main reason pip wants to copy files in the first place is to avoid accidental pollution between different builds using the same local tree; but if a build system implements a global cache like this then surprise, now you can get pollution between arbitrary builds using different trees, or between builds that don't even use a local tree at all (e.g. running 'pip install numpy==1.12.0' can potentially cause a later run of 'pip install numpy==1.12.1' to be corrupted). And, it assumes that all build systems can easily support out-of-tree incremental builds, which is often true but not guaranteed when your wheel build has to wrap some random third party C library's build system.
-- Nathaniel J. Smith -- https://vorpus.org