[Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Thu, Jul 6, 2017, at 07:19 PM, Paul Moore wrote:
> That's a good point - and provides a good contrast to my perspective
> as a pip developer that *pip* gets issues raised that aren't really
> pip's problem. I think it's in everyone's best interests to ensure
> that the user's experience is as unambiguous as possible in saying
> where any given problem lies.

Working on Jupyter, I've also seen plenty of misattributed issues from
people who think that we make the whole ecosystem they're using in the
notebook. I'm all for making it as unambiguous as possible, but I also
believe that in many cases it's impossible to be totally clear which
part has gone wrong, especially for users unfamiliar with the stack. So
my priority is to minimise user-facing failures, because we're all
likely to get bug reports.

> One thought occurs to me in that context - in my view, we should be
> clearly presenting to the user that it's *pip's* role to do the
> install, and flit's responsibility is to build wheels. I know that
> flit includes an install command, but I view that as a temporary
> workaround for the fact that PEP 517 isn't implemented yet. I'd be
> interested to know if you agree with that.

Yes-ish. There are three parts:

- flit installfrom (e.g. to install from a Github repo): entirely a
stopgap measure, I'm very happy to pass this responsibility over to pip.
- flit install (local install): I'll probably recommend pip once that
works, but may leave it in place a bit longer. It already works by
building a wheel and asking pip to install it.
- flit install --symlink (development install): stays around at least
until there's a standardised approach for this that I'm happy with.

> But essentially, we're promoting "pip install
> <whatever>" as the canonical install command, and "pip wheel
> <whatever>" as the canonical "build a wheel" command - backend
> specific commands should be for specialised use only, as I see it.

Depending on exactly what you mean by 'specialised'. I don't see flit as
'a PEP 517 backend', but rather 'a tool which provides a PEP 517
backend'. I will continue to recommend that developers invoke flit
directly to build and publish packages, but it should be transparent to
typical downstream users.

> Personally I wouldn't have a major problem with this, although I don't
> think Donald would agree, as there's questions that he's raised around
> potential inconsistencies between sdists and wheels built direct from
> the source tree that are unanswered in this model. My biggest concern,
> though, is that if we take this view, then it's critical that we have
> a reliable and efficient means of *copying* source trees.

So we have two alternative proposals for this bit:

1. Try to make an sdist, fall back to copytree if not supported
+ Provides some measure of built-in checking for the sdist problem
+ Reuses existing sdist machinery
- Fallback may be slow

2. Separate hook for efficient copy
+ Single mechanism is more predictable than primary+fallback
+ Reliably efficient
- Requires extra backend code

I'm willing to implement the necessary for either (but preferably not
both!). I think 2 is perhaps a bit more user friendly - it's not going
to be inexplicably slow because you've hit the fallback case.

> Tox may have more stringent requirements - currently it requires the
> ability to build a sdist to install from, and I'm inclined to think
> that this is a deliberate design choice rather than merely a
> convenience. I'm guessing that no-one has particularly explored the
> question of how tox would interact with flit-based projects yet?

I don't think so.

> Would
> it be acceptable to say that tox only works on a full checkout with
> VCS tools present (i.e, what flit needs to build a sdist) for
> flit-based projects? I don't really know.

I'd be willing to explore whether we can do better than that, but I see
tox as a developer tool, so I wouldn't consider it a show-stopper if it
required the presence of a VCS.

Thomas