[Distutils] Fwd: Re: PEP 517 again

[Paul Moore]

On 25 August 2017 at 09:38, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Fri, Aug 25, 2017, at 08:47 AM, Paul Moore wrote:
>> I've never cared about whether the backend modifies the source tree,
>> only that the direct-wheel / sdist->wheel equivalence holds.
>
> If a user types 'pip install .' in a directory where they don't have
> write permissions, what should happen?
>
> 1. The backend is capable of building without writing in that directory.
> 2. The frontend (pip) notices and pre-emptively copies to a writable tmp
> directory.
> 3. The user accepts that the operation may fail, depending on the
> backend.
>
> I'd go for option 1 or 3.

I'd go for option (1). I don't think we want to get into (2), with the
frontend trying to spot possible issues. I think (3) is always a
possibility, but users have a right to expect that backends will do
whatever they can to avoid it.

I'd *also* expect that the wheel built (and hence what gets installed)
would be the same as what you'd get if the user added write
permissions and then ran the command. That's part of what I mean by
direct wheel and sdist->wheel being equivalent. I'd completely support
any user who complained that getting different results in those 2
cases is a bug in the backend. But as Nick pointed out, this is
probably more detail than we need in the PEP.

> Here's a proposed update to the build_wheel section of the PEP:

(Apologies if I miss a change here, I don't have the original copy
right now to compare against).

> build_wheel(wheel_directory, config_settings=None,
> metadata_directory=None):
>     ...
>
> Must build a .whl file, and place it in the specified wheel_directory.
> It must return the basename (not the full path) of the .whl file it
> creates, as a unicode string.
>
> If the build frontend has previously called
> prepare_metadata_for_build_wheel and depends on the wheel resulting from
> this call to have metadata matching this earlier call, then it should
> provide the path to the created .dist-info directory as the
> metadata_directory argument. If this argument is provided, then
> build_wheel MUST produce a wheel with identical metadata. The directory
> passed in by the build frontend MUST be identical to the directory
> created by prepare_metadata_for_build_wheel, including any unrecognized
> files it created.
>
> Backends which do not provide the prepare_metadata_for_build_wheel hook
> may either silently ignore the metadata_directory parameter to
> build_wheel, or else raise an exception when it is set to anything other
> than None.
>
> Wheels built by this hook should, as far as possible, be equivalent to
> first building an sdist and then building a wheel from that sdist.

This is the point that I care about.

> Frontends which wish to ensure this consistency may call build_sdist
> (see below) first, and then call build_wheel in the unpacked sdist. But
> if build_sdist fails with NotImplementedError, they will fall back to
> calling build_wheel in the source directory, and rely on the backend to
> produce an equivalent wheel.

This seems to contradict the previous point - I'd rephrase it as
"Frontends which want to protect against backends violating this
expectation may call build_sdist..." and at that point it starts to
seem like excessive paranoia on the part of frontends.

I repeat my earlier comment here - do we have any actual cases where
we expect in-place wheel builds to *not* enforce the equivalence? This
was originally a huge debate issue because of all the comments about
development builds and incremental compilation, and caching. Are the
people who insisted that backends be allowed to carry build artifacts
between compiles now accepting that they have to ensure build
consistency? Or have they just dropped out of the discussion? I'm
pretty sure Nathaniel was one of those people, but I'm equally sure
he's fed up of debating theoretical possibilities. I suspect he still
considers my assumptions fairly naive, though, as I have no experience
of the sort of complexities involved in building the scientific
stack...

Either way, pip *can* go sdist->wheel, but as things are worded here,
I don't see a real reason to bother. I'd be more inclined to push any
bug reports about getting different results to the backend, quoting
the PEP's requirement for build equivalence, and only implement
sdist->wheel in pip if the backends pushed back (at which point, we'd
need to reopen the debate on what the statement in the PEP about
equivalence actually meant... )

> The source directory may be read-only. Backends should therefore be
> prepared to build without creating or modifying any files in the source
> directory, but they may opt not to handle this case, in which case
> failures will be visible to the user. Frontends are not responsible for
> any special handling of read-only source directories.

This is a useful addition, particularly as it's a general point, not
tied to any specific backend implementation.

Personally, I'd expect a backend to do its best to avoid failing - for
example by copying the files to a temporary location and doing a build
there - but that's reasonably considered a backend quality of
implementation issue.

> The backend may store intermediate artifacts in cache locations or
> temporary directories. The presence or absence of any caches should not
> make a material difference to the final result of the build.

I'm fine with this point. I'd also be fine with noting that such cache
locations can be stored in the source tree if the backend wishes to do
so (in which case, the sdist->wheel route is effectively a way of
deleting any such caches). But that's probably unnecessary.

Overall, this looks good to me.

Paul