[Distutils] Installed Extras Metadata

[Nick Coghlan]

On 11 February 2018 at 19:17, Nathaniel Smith <njs at pobox.com> wrote:

[snip a potential plan for migrating to extras-as-virtual-packages]

> - When pip needs to find a wheel like 'requests[security]', then it
> first checks to see if this exact wheel (with the brackets) is
> available on PyPI (or whatever package sources it has available). If
> so, it uses that. If not, then it falls back to looking for a
> 'requests' wheel, and if it finds one, and that wheel has 'extra'
> metadata, then it *uses that metadata to generate a wheel on the
> spot*, and then carries on as if it had found it on PyPI.

It occurs to me that this synthetic metadata generation aspect could
be pursued first, since these "installed extras" pseudo-packages
should be relatively straightforward to create:

- dist-info directory derived from the corresponding install package
name plus the extra name
- a generated METADATA file using the "name[extra]" pseudo-name and
amended dependency spec
- an appropriate RECORD file
- INSTALLER and REQUESTED files as defined in PEP 376

While we could potentially use a different suffix for the synthetic
pseudo-packages (e.g. "{name}[{extra}]-{version}.extra-info"), such
that anything looking only for "*.dist-info" packages would ignore
that, we could also decide that the non-standard name structure was
enough of a distinguishing marker.

Once we had install-time generation of those, then we could
*separately* consider if we wanted to allow extras to expand beyond
their current "optional dependencies" capabilities to also cover the
installation of additional files that aren't in the base distribution,
without requiring those files to be moved out to a separate
dependency.

The benefit of that split approach is that the "Database of installed
extras" aspect would only requires enhancements to pip and other
installers, and to tools that read the installation metadata, while
the full proposal would also require changes to PyPI and to publishing
tools.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia