[Distutils] A possible refactor/streamlining of PEP 517

[Paul Moore]

On 6 July 2017 at 22:54, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>
> I think Paul & Donald have been pretty adamantly against trusting backends
> to build tidily, though

On reflection, I'm less concerned about this than I was. If you wanted
to propose a stripped down version of PEP 517 which assumed it was the
backend's responsibility to ensure reproducible isolated builds, I'd
be willing to listen. But the proposal would need to include some
pretty strong requirements on precisely what we're asking of backends
- if build isolation is pip's problem to solve, then I'm happy for us
(the pip devs) to take that responsibility, and agree hooks that we
need to do so, but if we're assuming backends handle it for us, I
think we need to document clearly what we're assuming (because
frankly, the pip devs are the ones with the experience of the
potential issues).

My concern here is the same one as always. People raising issues where
they get a failed install, and it's because there was some sort of out
of date artifact in the source directory. No-one *expects* this to
happen, but we (the pip devs) are a bit paranoid about the possibility
because we've had to deal with so many "this shouldn't have happened"
issues around builds in the past. Admittedly, those issues are
typically with the arcane depths of setuptools, and with projects that
are being overly clever in setup.py, but it has left us with a bad "if
someone can mess things up, someone will" attitude.

I'm reasonably comfortable that a backend like flit that is limited to
pure-python projects, should be OK to ensure clean builds. There's not
*much* you can do to mess up zipping up some Python files. But once C
compilers and the like get added to the mix, it gets much harder to
ensure that a stray object file in the wrong place doesn't mess things
up. Add some of the really nasty stuff that the scientific people seem
to need, and I've no idea how anyone could guarantee clean builds
without creating a brand new tree. And yet the scientific people are
also the ones who want inplace incremental builds to mitigate their
really long build times. So there's pressure there to *not* copy for
the sake of faster builds.

So I'm not as against trusting backends as I was. But I am concerned
that we make sure that the backend authors clearly understand the
problem they are responsible for (and this is a case where flit is
*not* a good example of a backend, as it avoids the nasty cases by
design).

Paul