[Distutils] Provisionally accepting PEP 517's declarative build system interface

[Paul Moore]

On 29 May 2017 at 08:05, Nathaniel Smith <njs at pobox.com> wrote:
> Right now pip doesn't really have a good way to expressing the latter.
> 'pip install directory/' is relatively unambiguously saying that I
> want a local install of some potentially-locally-modified files, and
> while it might involve a temporary wheel internally there's no need to
> expose this in any way (and e.g. it certainly shouldn't be cached), so
> I think it's OK if this builds in-place and risks giving different
> results than 'pip install sdist.tar.gz'. (Also note that in the most
> common case where a naive user might use this accidentally, where
> they've downloaded an sdist, unpacked it manually, and then run 'pip
> install .', they *do* give the same results -- the potential for
> trouble only comes when someone runs 'pip install .' multiple times in
> the same directory.)

I think that the key thing here is that as things stand, pip needs a
means to copy an existing "source tree", as efficiently as possible.
For local directories (source checkouts, typically) there's a lot of
clutter that isn't needed to replicate the "source tree" aspect of the
directory - but we can't reliably determine what is clutter and what
isn't.

Whether that copying is a good idea, in the face of the need to do
incremental builds, is something of an open question - clearly we
can't do something that closes the door on incremental builds, but
equally the overhead of copying unwanted data is huge at the moment,
and we can't ignore that.

Talking about a "build a sdist" operation brings a whole load of
questions about whether there should be a sdist format, what about
sdist 2.0, etc into the mix. So maybe we should avoid all that, and
say that pip[1] needs a "copy a source tree" operation. Backends
SHOULD implement that by skipping any unneeded files in the source
tree,but can fall back to a simple copy if they wish. In fact, we
could make the operation optional and have *pip* fall back to copying
if necessary. It would then be a backend quality of implementation
issue if builds are slow because multi-megabyte git trees get copied
unnecessarily.

(This operation might also help tools like setuptools-scm that need
git information to work - the backend could extract that information
on a "copy" operation and pit it somewhere static for the build).

Paul

[1] As pip is currently the only frontend, "what pip needs right now"
is the only non-theoretical indication we have of what frontends might
need to do, so we should be cautious about dismissing this as "pip
shouldn't work like this", IMO.