[Distutils] PEP 517: Open questions around artifact export directories

Nick Coghlan ncoghlan at gmail.com
Mon Jun 12 21:27:34 EDT 2017

On 13 June 2017 at 08:55, Nathaniel Smith <njs at pobox.com> wrote:
> On Mon, Jun 12, 2017 at 3:34 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>> But honestly, I think we're at the point where someone just needs to
>> make a decision - there's very little compelling evidence either way.
> I was the original PEP author, but Thomas has mostly taken it over at
> this point, so I'm not sure how much you should listen to me :-). But
> if you pointed at me and told me to make some decisions, then right
> now this is what I'd do:
> 1) Go back to having the wheel generation hook generate a .whl file
> Rationale: the optimization benefits of generating an unpacked wheel
> are unclear, but we know that reproducible builds are important, and
> filename encoding is tricky and important, and that having a common
> well-understood standard with tooling around it is important, and on
> those axes .whl unambiguously wins. And if there do later turn out to
> be compelling optimization benefits to generating unpacked wheels,
> then we can add an optional generate_unpacked_wheel hook in the
> future.

Despite being the one to originally propose standardisation on passing
directory paths around, I'm starting to lean back towards this

My rationale for this doesn't really have a lot to do with topics
we've discussed so far, and instead asks the question: what would work
best for an installation frontend that wanted to keep the actual build
tools off the system doing the installation, while still allowing for
transparent "from sdist" installations?

And that's where cross-platform archive formats really shine: since
they're incredibly convenient for network transfers, they make fewer
implicit assumptions about *where* the work is being done. Whereas if
we rely on directories in the baseline interface specification, we're
making a lot of assumptions about how all future frontends are going
to work.

It should also be relatively straightforward to add cross-platform
validators to frontends that check that archives are well-formed, but
it's harder to write such validators that work cross-platform on
arbitrary directories.

> 2) Specify that the wheel generation hook, metadata hook, and sdist
> hook return the name of path that they created as a unicode string.
> Rationale: fixes a point of ambiguity in the current spec.

And still leaves the door open to supporting multiple wheels in the
future by returning a list instead of string.

> 3) Go back to having the sdist generation hook generate an archive
> file instead of an unpacked directory
> Rationale: Essentially the same as for point (1). The additional
> complication here is that we know that pip plans to use this as part
> of its standard build-from-unpacked-source pipeline, so if we're
> trying to optimize this case for developers with their fast SSD drives
> etc. then the compress/decompress cycle actually does matter.

I think you also raise a good point in bringing up the "First make it
work, then make it right, then make it fast" priority order.

We're currently in the "make it right" phase, and I think you're
correct that that's much easier to achieve cross platform with archive
based data exchange - otherwise we run the risk of running into
surprises with NTFS, HDFS+, etc (let alone folks running builds over

So the core "make it right" API would be "build_sdist" and
"build_wheel" as mandatory backend hooks, with
"prepare_wheel_metadata" and "prepare_build_files" as optional "make
it faster" helpers.

We'd then reserve "prepare_sdist_content" and "prepare_wheel_content"
as possible future "make it faster" APIs, rather than adding them now.

> 4) Specify that the sdist archive file must be .tar.gz
> Rationale: we only need one and it reduces variation, and dstufft
> likes .tar.gz better than .zip.


> 5) Drop the prepare_files hook
> Rationale: it's purpose is somewhat unclear at this point, and it
> gives up the main advantage of going through sdist (the guarantee that
> building from sdist and building from unpacked tree give the same
> results) while still having the main disadvantages (you have to copy
> everything and lose incremental rebuilds).

The main reason this made it into the PEP is that build_sdist may have
additional dependencies that prepare_build_files doesn't (e.g. Thomas
has indicated that flit needs to call out to VCS tools to build a full
sdist, but can export its own input files directly).

So while it's technically a "make it faster" hook rather than an
essential "make it right" hook, I think just smoothing out the
interaction between pip & flit specifically provides sufficient value
to make it worth including in the initial version of the


Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Distutils-SIG mailing list