[Distutils] PEP 517: Open questions around artifact export directories

[Nick Coghlan]

Hi folks,

Thanks for the recent discussions around the PEP 517 build hooks PEP -
I think they've clarified and improved the PEP in several areas and
are setting us up well for an interoperability API that is both
flexible and straightforward for frontends and backends to support.

However, I think they've also raised some new still open questions,
which I'd like to discuss specifically in this thread.

1. Should backends create artifacts directly, or simply export their
contents, and let frontends assemble them into redistributable
archives?
2. Are backends responsible in any way for naming artifacts, or should
that responsibility lie entirely with frontends?

For the first question, in the latest PEP draft, we have:

- the sdist hook is `export_sdist`, with the expected output being a
filesystem directory
- the metadata hook is `get_wheel_metadata`, with the expected output
being the contents of the metadata directory
- the wheel hook is `build_wheel`, with the expected output being an
already assembled binary wheel archive
- the build preparation hook is `prepare_build_files`, with the
expected output being the subset of files specfically needed to
assemble a full wheel file with `build_wheel`

For the second question:

- the sdist hook is currently unclear on whether the backend should
write the sdist contents directly into the frontend supplied
directory, or into a suitably named subdirectory
- the metadata hook is explicit that the backend should write the
metadata directly into the given directory
- the wheel hook is explicit that the backend should create already
assembled wheel archives in the given directory
- the build preparation hook is explicit that the backend should write
the metadata directly into the given directory

When considered as a unified specification this seems confusingly
inconsistent to me, although understandable given that we started out
covering only the well specified wheel format, and eventually expanded
it to cover the substantially less well specified notions of sdist
archives and temporary build directories.

While I'm open to being persuaded otherwise, my current thinking is
that these concerns could be suitably addressed by making the
following amendments:

1. Move all artifact naming responsibilities to the frontend. All
backend hooks would just write files in the defined format into the
directory supplied by the frontend (and those directories would have
no defined naming scheme - they're just a staging area for the backend
to pass content to the frontend)

2. Change the names of the artifact hooks to be "export_sdist_content"
and "export_wheel_content" to emphasise that backends are responsible
for generating the artifact *contents*, not the artifacts themselves

3. Change the optional metadata preparation hook to be
"prepare_wheel_metadata" for consistency with "prepare_build_files".
This gives us a consistent naming scheme where the "export_*" hooks
are required, but each has an optional "prepare_*" counterpart that
backends may implement as a more efficient alternative when the full
artifact contents aren't (or may not be) needed.

Assuming we add support for building multiple wheels from a single
source archive in the future, we'd do this through some kind of
declarative API based mechanism like adding a new optional
"wheel_name" parameter to `prepare_wheel_metadata` and
`export_wheel_content` (to tell the backend which wheel the frontend
is interested in), along with an additional field in [build-system] to
tell frontends which secondary wheels are defined by that project.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia