[Distutils] Finishing up PEP 517

[Nick Coghlan]

On 17 June 2017 at 08:05, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Fri, Jun 16, 2017, at 10:48 PM, Nathaniel Smith wrote:
>> The messy complications come from
>> prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't
>> surprising, since those are the two hooks where we're squinting into
>> our crystal ball to try and guess what will be useful for software
>> that doesn't exist yet, but will later, maybe, we hope.
>
> I'm not exactly clear on what use cases the prepare_wheel_metadata hook
> satisfies - that was in the spec before I was particularly involved in
> it.

As Donald notes, this is primarily a workaround for the
underspecification of the sdist format - wheel metadata is the most
trustworthy information source we have for post-installation project
metadata (name, version, runtime dependencies, etc), since it's
exactly what will end up in the post-install dist-info directory
(minus the files generated by an installer at installation time).

Pure-Python backends like flit can probably skip providing it, since
building a wheel is a cheap operation, but something like enscons is
probably going to want to implement it (since only generating the
Python wheel metadata should be much cheaper than running the full
build).

And unlike defining a new iteration of the sdist format, encouraging
backend developers to make wheel metadata generation a cheap operation
doesn't require any new speculative format definition work on our
part.

> I do think we've hashed out a concrete need for prepare_build_files (or
> whatever it ends up called): to copy the files to a build directory
> without either copying large amounts of unnecessary data or figuring out
> what belongs in an sdist. The only alternative would be to require
> backends to avoid leaving clutter in the working directory, so they
> would take care of any necessary copy step rather than exposing a
> separate hook. People insist that trusting backends is a non-starter,
> though.

It isn't that trusting them is a non-starter in all situations, it's
that *requiring* front-ends to trust them is a non-starter.

Instead we want front-ends to have the following expectation:

1. either a backend's build_sdist is always fast, and doesn't require
any dependencies not already needed to build a wheel;
2. or else, if build_sdist isn't always fast, or requires extra
dependencies, then prepare_wheel_input_files will be provided

"Dependencies" there refers to both actual Python-level dependencies
expressible in terms of PEP 508, as well as external dependencies that
currently can't be represented (e.g. requiring particular command line
tools to be available).

Cheers,
Nick.

P.S. This also seems like an opportune time to remind folks that
Tennessee Leeuwenburg and Robert Collins started a draft PEP for
expressing external dependencies a couple of years ago:
https://github.com/pypa/interoperability-peps/pull/30/files

While any further work along those lines would need significant
updates to account for PEP 508, PEP 518, PEP 517, and other
developments, the core concept of using
"<dependency-type><delimeter><dependency-expression>" so that external
dependencies can potentially be passed anywhere that PEP 508
dependencies are currently supported remains sound, and would likely
provide a solid foundation for a plugin based model where (for
example), there might be common "bin", "clib", and "cheader" types at
the dependency declaration level (but platform dependent
implementations of those plugins), as well as cross-platform
namespaces for 3rd party dependency management ecosystems (e.g. npm,
cargo, maven) (which would then either have appropriate build system
requirements like "bin!!npm", "bin!!cargo" and "bin!!maven",
dependencies on Python wrappers for those tools, or else dependencies
on suitable extras in the underlying build backend).

The one place where we likely *wouldn't* mix internal and external
requirements is in interoperability specifications: there, we'd keep
the external requirements separate, so that older tools don't get
confused trying to parse them, and instead just treat satisfying them
as somebody else's problem (i.e. preserving the status quo).

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia